Ubuntu Server not completing upgrade

It’s been about seven months since I setup a Wireless GitLab server. Since then I’ve figured out how to list updatable packages on Ubuntu Server. I’ve also performed several updates using sudo apt-get update && sudo apt-get upgrade.

gzip: stdout: No space left on device

Today I ran into a new problem. Upon trying to perform an update I was presented with a peculiar error. It said gzip: stdout: No space left on device and it told me to run apt-get -f install to fix things up. So… that’s what I tried doing. I tried running the apt-get -f install command but to no avail. The command would not complete successfully.

This is about the time when I start getting really annoyed with Linux and the command line and all the things associated with configuring things manually like do I really need to download the entirety of the Linux MAN files inside my HEAD? DO I NEED TO DO THAT? GAHasldkjsadljfsadfsdsdf!!!!

Calm yourself.

The /boot partition is 100% full

Ok, so it turns out that the apt-get process can fail if the /boot partition becomes 100% full. There were a number of suggestions online that indicated you needed to clean out the /boot partition by removing old linux-images that you don’t need anymore. Many of these suggestions involved using sudo apt-get remove [package-name] or using sudo apt-get autoremove which are both completely valid options… IF APT-GET WERE WORKING. But apt-get is not working, that’s the problem.

So… I Googled a lot and dug through a lot of forums. Finally I stumbled on this uber helpful answer on askUbuntu. I’ll go ahead and paraphrase the answer below so that I can easily find it again. Yes. This is all about me.

Cleaning up the /boot partition

In the case where your /boot partition becomes totally full you can use these steps to clean it up. (From flickerfly on AskUbuntu).

  1. Run the following command to get a list of the linux-image files that you don’t need anymore.
    sudo dpkg --list 'linux-image*'|awk '{ if ($1=="ii") print $2}'|grep -v `uname -r`
    
  2. Create a command to remove the folders you don’t need anymore. You can do that with a command like this (where brace-expansion is used to save keystrokes). Use the output from the command above to build your command.
    EXAMPLE
    sudo rm -rf /boot/*-3.2.0-{23,45,49,51,52,53,54,55}-*
    
  3. Now that apt-get has space to work with you can actually run sudo apt-get -f install to clean things up.
  4. Use Purge to manually resolve issues with “Internal Errors” (if you get any internal errors).
    EXAMPLE
    sudo apt-get purge linux-image-3.2.0-56-generic
    
  5. Run `sudo apt-get autoremove ` to clean up anything orphaned by the manual clean.
  6. Now you can finally proceed with those updates you were wanting to do.
    sudo apt-get update && sudo apt-get upgrade
    

Party?

We can party now I think.

Secrets were required, but not provided

Psst… tl:dr -> rebooting my wireless router fixed the problem.

A few months ago I setup a wireless Gitlab server. This server has been working great. Once in a while I check up on it via SSH and make sure it’s updated. Otherwise, I leave it alone and it’s happy.

That is until today.

Secrets were required, but not provided

Today, for some reason, I could not access my gitlab server via the web interface or push to it via the git cli. In fact, I couldn’t even SSH into it. I had to pull out the ol’ physical monitor and keyboard and MANUALLY connect. Shudder.

First thing I do upon connecting to the server is try to ping google.com of course. It didn’t work. The server could not find the address for Google, and as anyone knows, if you cannot find Google then the internet does not exist. Plain and simple. You might as well be trying to fly a kite in the Marianas trench.

Now, up until this point I’ve had no issues connecting to the internet. The server automatically connects to the WIFI no big deal. However, I thought that maybe the network authorization expired or something? Maybe I can only connect for a few months at a time before re-authenticating. So I tried just that. I whipped out my old friend nmcli and ran:

nmcli device wifi connect MyAccessPoint password 123456789ACB

This is when I see the dreaded Secrets were required, but not provided response. Well – I’m not sure what secrets it wants me to tell it but I mean, the password was right, and I’m certainly not telling it who my favorite little pony is or even if I like little ponies.

I tried several more times, I tried rebooting the server. Nothing worked. It was at that point that I, using another computer, logged into my wireless router and told it to reboot. Several minutes later everything is fine.

Rebooting my router fixed the problem

Why? I don’t know. There has been some talk that some routers will auto select channels that some linux machines do not like. I think that was likely the original issue. Rebooting the router worked because another channel was selected. In any case… long story short. If you are having issues with your wireless server not connecting to your network then try rebooting your router.

Password Protect a WordPress Subdirectory with .htaccess

There are questions all over the internet regarding how to password protect a sub-directory when you are using WordPress.

I just spent a long time fighting a frustrating battle with this as well. So I’m documenting the resolution here for my (and anyone’s) benefit.

 In short

  1. WordPress does not mess with requests to actual directories or files.
  2. If WordPress is messing with your request then you aren’t requesting an actual directory or file.
  3. It’s likely your Error codes aren’t setup to return actual files.
  4. Make sure your .htaccess file isn’t generating 500 errors (i.e. ensure the path to your .htpasswd file is correct).

Problem

I’ve added a .htaccess and .htpasswd file but all I see is a WordPress 404 page. I can’t stop crying because it’s not working and my brain hurts.

Yep. That happens. WordPress comes with the following .htaccess file by default:

# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>
# END WordPress

Let’s break this down.  First we are checking if the mod_rewrite module is even installed. If it is then we are turning the RewriteEngine on. That’s all great. We wouldn’t want to use the engine if it didn’t exist… right?

RewriteBase / – This sets the base of every subsequent Rule and Condition to the root `/`.  This way we don’t have to include the root directory at the beginning of any of our rules.

RewriteRule ^index\.php$ - [L] – This rewrite rule checks to see if we are on the index.php page already. The dash in the rule means do nothing. So… if we are already on index.php don’t do anything. The [L] option means that we should stop processing rules now. Don’t do anything else, we’ve got what we wanted. Quite literally this is the [L]ast rule that should be processed.

RewriteCond %{REQUEST_FILENAME} !-f – This condition makes sure that if the current request is hitting an actual existing file then we should do nothing. So WordPress won’t mess with your requests if you try to link to an actual file.

RewriteCond %{REQUEST_FILENAME} !-d – This condition makes sure that if the current request is hitting an actual existing directory that we should do nothing. So WordPress won’t mess with your requests if you try to link to an actual directory.

RewriteRule . /index.php [L] – Finally, if our request passed the above two conditions (it’s not an actual file and not an actual directory) then map the request to index.php. Now the request is mapped and WordPress can do its thing!

That’s Great But…

I know what you are thinking. You are thinking:

If what you are saying is true, then I shouldn’t be seeing a 404 page. My password protected directory actually exists!

Yes. You are correct, your directory does exist.

Solution

When you password protect a directory with .htaccess you are telling the server to return a certain response code. The 401 response code meaning the user is unauthorized, to be precise. When the browser received this response code it triggers a username and password prompt. However, and here is the problem, the browser is never receiving the response code.

Why is the browser not receiving the response code?

Good question. If you remember the WordPress .htaccess checks if the requested url points an actual file or directory. It only rewrites you to the index.php file if you aren’t actually requesting a file or directory. When you throw the 401 response code you aren’t actually requesting a file or directory. You are essentially requesting nothing (because you are unauthorized). So the WordPress .htaccess file is behaving correctly – it’s rewriting you to the index.php page and giving you a 404 (because more than likely your password protected directory does not match a permalink on your WordPress blog).

So… if WordPress is making sure that you actually requested a file then… you need to make sure that you are actually getting a file! You can do this by adding the following line to the top of your WordPress .htaccess file:

ErrorDocument 401 default

What you are doing is telling the server to return the default 401 file when it encounters a 401 response code. Once you are returning an actual file WordPress won’t try to grab your request.

Ok. I added that and I’m still having issues. What gives?

If you are like me, then the 401 response code fix wasn’t enough. You are still having the same issue and by now you are wanting to… oh gosh I can’t even think of anything to describe this type of pain.

Let’s look at our .htaccess file we are using to password protect our sub-directory. If you are anything like me your file might’ve looked something like this.

AuthType Basic
AuthName "Password Protected Area"
AuthUserFile /public_html/jeremysawesome.com/mySecretDirectory/.htpasswd
Require valid-user

This looks perfectly valid to me. However, it turns out this file is generating Internal Server Errors!  (I know because I added a ErrorDocument 500 default line to my WordPress .htaccess file just for kicks.) But this shouldn’t be generating a 500 error unless I’m doing something wrong.

Turns out. I was.

The AuthUserFile argument needs to be the full server path to your .htpasswd file. Turns out, /public_html wasn’t actually the beginning of my server path. As a result the server was throwing a 500 error. Once I figured out what my entire full server path was, and added that to my .htaccess file, everything started working.

To Recap

  1. WordPress does not mess with requests to actual directories or files.
  2. If WordPress is messing with your request then you aren’t requesting an actual directory or file.
  3. It’s likely your Error codes aren’t setup to return actual files.
  4. Make sure your .htaccess file isn’t generating 500 errors (i.e. ensure the path to your .htpasswd file is correct).

Whew! Thank goodness that’s over. Happy Blogging 🙂

JavaScript Scoping. Callbacks and Loops

I just ran into this issue last night. The problem: I had a loop that was adding a callback to a method. Something like this:

for(var i=0;i<10;i++){
    $myElement.on('some-event', function(){
        DoSomethingWith(i);
    });
}

What I expected was that the value of the i variable at the time it was called would be used in my callback method. However, this was not the case… the i variable was the same in every single callback.

See this JSFiddle for an example.

The reason for this? JavaScript variable hoisting.  Before your code is executed it is scanned and the variables are processed. This has the effect of moving your variables to the top of the current function regardless of where in the function they are defined. (Except for in cases where you are implicitly declaring global variables).

So, in our situation we’ve defined var i. This is processed before the loop is processed and it is as if we wrote this:

var i;
for(i=0;i<10;i++){
    $myElement.on('some-event', function(){
        DoSomethingWith(i);
    });
}

Now it becomes a bit more clear why we are running into the issue with i being the same. The reason is because by the time the callback is executed the for loop has already run and the value of the i variable is already 10.

The solution, as far as I can tell, is to use an IIFE to scope the variable correctly in order store the current value for later. It looks ugly and it feels hacky… but it seems to be what is necessary. Update: It appears that you can also use .bind to set the value correctly as well.

var i;
for(i=0;i<10;i++){
    $myElement.on('some-event', 
        (function(i){
            return function(){
                DoSomethingWith(i);
            };
        })(i);
    );
}

And the JSFiddle to demonstrate.

Example With .bind

var i;
for(i=0;i<10;i++){
    $myElement.on('some-event', DoSomethingWith.bind(undefined, i));
}

Deserializing JSON to a .NET Object

Generally in .NET MVC you would use the default model binders to deserialize JSON to a .NET object. However, there are some cases, involving “complex” collections, where this becomes a bit tedious to do. Microsoft provides a few ways around this, but none are satisfactory. Unlike the default model binders, the JavaScriptSerializer provides support for deserializing JSON to a .NET object with complex collections.

Take the following simple class and method signature for example:

// simple class
public class SaveInformation
{
    public string Name { get; set; }
    public Dictionary<string, List<SaveItem>> Components { get; set; }
}

// method signature
public ActionResult Save(SaveInformation saveInformation)

Below I’ve included some simple JSON that we will pass to .NET via AJAX:

{"SaveToSession":false,"Name":"My Stupendous Thing","Components":{"Component1":[{"ProductId":"1234"}]}}

We are passing the JSON to .NET using AJAX with the “application/json” contentType. Because we are using the “application/json” contentType, .NET will automatically call the JsonValueProviderFactory and map the information over to our SaveInformation model. It seems pretty straight forward, the Save method will receive a SaveInformation object when it is called.

Not so straightforward unfortunately, the Save method does receive a SaveInformation object. If we inspect our SaveInformation object we see the Name is populated perfectly fine, but the Components dictionary ends up being null.

The reason for this? It seems that the JsonValueProviderFactory doesn’t fully support pure JSON syntax. In order for .NET to properly parse a complex collection, you actually have to give it a numerical index.

From http://msdn.microsoft.com/en-us/magazine/hh781022.aspx:

Though it’s somewhat counterintuitive, JSON requests have the same requirements—they, too, must adhere to the form post naming syntax. Take, for example, the JSON payload for the previous UnitPrice collection. The pure JSON array syntax for this data would be represented as:

[
    { "Code": "USD", "Amount": 100.00 },
    { "Code": "EUR", "Amount": 73.64 }
]

However, the default value providers and model binders require the data to be represented as a JSON form post:

{
    "UnitPrice[0].Code": "USD",
    "UnitPrice[0].Amount": 100.00,

    "UnitPrice[1].Code": "EUR",
    "UnitPrice[1].Amount": 73.64
}

I know right? It was hard for me to read this as well due to the anime-sized tears in my normal-sized eyes. The problem is, no browser I know of has a JSON.serializeForDotNet function. So this leaves us with two options, make our own JSON serializer, or make our own ValueProvider/ModelBinder. Neither of these options sounded very appealing to me, so I went with the hidden third option. Pass in a string 🙂

You can use the JavaScriptSerializer class, located in System.Web.Script.Serialization, to deserialize JSON information and, as an added bonus, it actually handles properly formatted pure JSON!

So, the new method looks like this. We pass in a string value and deserialize it using the JavaScriptSerializer. Now we receive a SaveInformation object with a fully populated Components dictionary.

public ActionResult Save(string saveInformation)
{
    // instantiate a JavaScriptSerializer
    JavaScriptSerializer serializer = new JavaScriptSerializer();

    // use the JavaScriptSerializer to deserialize our json into the expected object
    SaveInformation saveInformationObject = serializer.Deserialize<SaveInformation>(saveInformation);

The Case of the Disappearing Cookie

If you are working on a site that doesn’t have a proper TLD (i.e. http://localhost:8013) you will notice that setting cookies on the browser doesn’t seem to work the way it should. Sometimes your cookie, which works perfectly fine on a site with a TLD, will not show up on your local site.

For example, this code will work perfectly fine on a site with a TLD but not on your local site.

newCookie = New HttpCookie(ConfigurationManager.AppSettings("YOUR_SPECIAL_COOKIE_NAME"))
newCookie.Domain = ConfigurationManager.AppSettings("BASE_DOMAIN")
HttpContext.Current.Response.Cookies.Add(newCookie)

At this point you might be thinking, ‘I did everything right, where did my cookie go?‘.

This comes down to how the browser handles setting cookies that have a specified domain. The browser prepends a ‘.’ (that’s a dot) to the beginning of any domain specified for a cookie. On normal domains this basically just says that this cookie is valid for this domain and any subdomains. The following non-doctored image proves it:

Set two cookies, one with a specified domain and one without.

Setting Test Cookies
JavaScript to set test cookies for the superfrogadventures.com domain.

This results in the following two cookies

Shows the results of the first cookie set.
Shows the results of the first cookie set.
The Second Test Cookie
Shows that the domain does not have a dot.

You will notice that the domains are a bit different. One has a ‘.’ while the other does not.

So. All this to say, if you are developing locally (or on a site without a TLD) then you will need to save your cookies without a “domain”. This will ensure that the cookie is accessible. Otherwise you will probably need to create yourself some sort of domain with a TLD.