6

I've tried the following, but I can't get a build date later than:

Tue, Aug 21 05:18:46 UTC 2012

I have done the following:

apt-get dist-upgrade
apt-get update
apt-get upgrade openssl

and

apt-get purge openssl
apt-get install openssl

and 

apt-get purge libcrypto1.0.0
apt-get install libcrypto1.0.0

Everything seems to work fine, but the build date remains as above. And the http://filippo.io/Heartbleed/ test is still failing.

I know I'm not crazy, because I was able to update my identical server Wednesday. (Identical in EVERY way except hardware).

EDIT:

I compared the /etc/apt/sources.list files on both machines and they appear to be identical. How did one server update and the other won't?

EDIT:

Did as suggested:

apt-get purge openssl
reboot
apt-get install openssl
reboot

to no avail.

Tried same commands on libssl1.0.0, still same version as listed above.

This one's got me stumped.

Any suggestions?

EDIT

As soon as I get enough street cred (15), I will +1 the viable workarounds

EDIT

As suggested, I ran apt-get with --reinstall --print-uris and got back:

http://us.archive.ubuntu.com/ubuntu/pool/main/o/openssl/openssl_1.0.1-4ubuntu5.12_amd64.deb

Then rebooted, same version as listed above. Still failing heartbleed.

user1182988
  • 162
  • 1
  • 1
  • 8
  • Did you reboot? Old/vulnerable copies will remain in use/memory so long as the processes using them do. – Maxx Daymon Apr 12 '14 at 22:41
  • try rebooting. and try getting the `deb`s off of the ubuntu packages site or something like that. Also, you can upvote on your own questions, and leave comments too, even below the points barrier. – Wyatt Ward Apr 13 '14 at 02:37
  • What is reported for `dpkg-query --list libssl1.0.0` next, what is "Filename:" from `apt-cache show libssl1.0.0` next, does that match the file you get when you `apt-get download libssl1.0.0`? – Maxx Daymon Apr 13 '14 at 08:57
  • Also helpful, `apt-get clean` to clear your download cache, then `apt-get install openssl libssl1.0.0 --reinstall --print-uris` and check the download URIs. Download the packages manually to inspect. – Maxx Daymon Apr 13 '14 at 09:11
  • Thanks Maxx, I ran the dpkg-query and apt-get download, and the versions match. I then ran apt-get clean, downloaded the packages and inspected their content, which matched the dpkg-query versions. Does that mean the repository is broken? – user1182988 Apr 13 '14 at 13:42
  • @user1182988 The repository looks good from here. I downloaded and verified the packages and they were correct. You can manually download the libssl and openssl deb packages and install them, but it looks like apt is selecting and installing the patch. Does your sha1sum for `/usr/bin/openssl` match `22297ff89bbdfd4befb6f66f79e41210768aa81a` and `/lib/x86_64-linux-gnu/libssl.so.1.0.0` match `4fbb25a3e82d3d835c68421a6c4647afe534b393`? What server(s) are you using, and is there a chance you have a statically linked libssl in it/them? – Maxx Daymon Apr 13 '14 at 20:53
  • @user1182988 Another test you can do is to download the .deb package and run `debsums libssl1.0.0_1.0.1-4ubuntu5.12_amd64.deb` to verify that the files you have installed match the sums from the known good package. Do a `dpkg -i filename.deb` to install manually. OpenSSL *should* return `OpenSSL 1.0.1 14 Mar 2012, built on: Mon Apr 7 20:33:29 UTC 2014` Your build date implies that the package is not getting installed. Check dpkg and apt logs for errors? – Maxx Daymon Apr 13 '14 at 21:14
  • @MaxxDaymon, Thanks for helping. I ran sha1sum on openssl, and the checksum matches the one you provided, however the libssl.so.1.1.0 did not match. I compared to my "identical" server, and both checksums matched. I searched my dpkg and apt logs, but saw nothing that looked like an error. The servers are typical LAMP with postfix and dovecot, and associated spam/security apps. I was about to remove libssl1.0.0, but it told me that there were too many things relying on it, and that I should reconsider unless I "Really know what I'm doing", so I reconsidered. will a dpkg-reconfigure fix it? – user1182988 Apr 13 '14 at 22:45
  • @user1182988 Can you do a `apt-cache policy libssl1.0.0` and tell me what your Installed: and Candidate: lines are? – Maxx Daymon Apr 13 '14 at 23:11
  • @user1182988 I think I would recommend a `wget http://us.archive.ubuntu.com/ubuntu/pool/main/o/openssl/libssl1.0.0_1.0.1-4ubuntu5.12_amd64.deb` followed by a `dpkg --force-all --remove libssl1.0.0` and then a `dpkg --force-overwrite -i libssl1.0.0_1.0.1-4ubuntu5.12_amd64.deb` (You could also do a plain `apt-get install libssl1.0.0` instead, eliminating the need to download the deb) I just tried both approaches on a 12.04 test server and they both worked. – Maxx Daymon Apr 13 '14 at 23:16
  • @MaxxDaymon, you nailed it. I ran `dpkg --force-all --remove libssl1.0.0` followed by `apt-get install libssl1.0.0`, and now my build is 4/7/14. I learned that dpkg will remove a package without removing all of the stuff that depends on it (right?). Thanks Maxx. Make an answer out of this and I will check it. – user1182988 Apr 14 '14 at 00:34
  • @user1182988 Fantastic! I'm stepping out for a few hours, I'll write it up in the form of an answer when I return. Glad to hear you're back in business! – Maxx Daymon Apr 14 '14 at 01:38
  • @user1182988 I posted an answer. It was a bit wall-of-text, so I reduced it to the essentials since we've got a lot of the diagnostics steps in the question now. – Maxx Daymon Apr 14 '14 at 06:22

4 Answers4

8

Forcing a package re-installation

Apt thinks that the packages are installed and upgraded. Manual verification and testing of libssl suggests otherwise, so the package database is inconsistent with the installed files (maybe files were previously upgraded previously without package manager involvement). For whatever reason this happened, the package is not being correctly upgraded or reinstalled. This assumes that it has been established that the system is reporting fixed versions, but is still showing as vulnerable.

First, attempt to forcibly reinstall the affected packages:

apt-get install --reinstall libssl1.0.0

If that fails, try forcing the complete removal of the package without involving any of Apt's dependency management and sanity checks:

dpkg --force-all --remove libssl1.0.0

At this point, the system is effectively 'broken' because libssl is missing and many packages are still installed that depend on it (this is what Apt tries so hard to prevent, and the reason we are going behind Apt's back), so reinstall libssl1.0.0, re-downloading the latest package from the repository:

apt-get clean && apt-get install libssl1.0.0

Alternatively, if you have downloaded the known good deb package, you can use dpkg to install and force overwrite of any existing files:

dpkg --force-overwrite -i libssl1.0.0_1.0.1-4ubuntu5.12_amd64.deb

Re-test and check (debsums, sha1sum) the installed files against known good configuration.

Maxx Daymon
  • 1,381
  • 7
  • 13
2

If your apt-get repositories don't contains any precompiled 1.0.1g OpenSSL version, so just download sources from official website and compile it.

Below the single command line to compiling and install the last openssl version.

curl https://www.openssl.org/source/openssl-1.0.1g.tar.gz | tar xz && cd openssl-1.0.1g && sudo ./config && sudo make && sudo make install

Replace old openssl binary file by the new one via a symlink.

sudo ln -sf /usr/local/ssl/bin/openssl `which openssl`

You are all good !

# openssl version should return
openssl version
OpenSSL 1.0.1g 7 Apr 2014

Cf this blog post.

NB: As stated in the blog post, this workaround will not fix "Nginx and Apache server who have to be recompile with 1.0.1g openSSL sources."

KoKo
  • 1,646
  • 2
  • 14
  • 24
  • Yes that's a great and simple workaround. The thing is, I know the binaries are available since the upgrade worked on Wednesday for my identical server. I thought perhaps that Ubuntu pulled it from the repository. – user1182988 Apr 12 '14 at 14:00
0

Maybe you have 2 versions of OpenSSL in your PATH. This can happen, if you compiled your own version.

Try this /usr/bin/openssl version.

This schould be the position of the package version of OpenSSL.

Peter Lamby
  • 442
  • 3
  • 8
  • Thanks for the reply. I did what you suggested (with -a) an got the same build date response. I have avoided compiling my own version of anything, as I need these servers to be easily-reproducible. – user1182988 Apr 12 '14 at 15:35
  • @user1182988 - I assume transferring the files from the working sever isn't an option? – Ramhound Apr 12 '14 at 17:23
  • Transferring the files is a viable option, but I'm concerned that the apt-get won't work. – user1182988 Apr 13 '14 at 02:15
0

You need to update libssl1.0.0, then reboot your server. It will not work without a reboot.

ITProStuff
  • 399
  • 1
  • 7
  • Thanks for the reply! I tried apt-get purge libssl1.0.0, reboot, apt-get install libssl1.0.0, reboot - same version as above. Tried the same thing using openssl, still same version and build mentioned above. This is weird! – user1182988 Apr 13 '14 at 02:10