Matt Wetmore

Installing Elm packages with a bad connection or VPN

Posted on July 17, 2015
  1. There are some pros and cons to this approach, as I see it. On one hand, package names are nicely namespaced by the authors’ names, solving the problem package managers such as npm have with nice package names getting taken early. Also, as explained in the announcement post, hosting on Github means offloading much of the necessary functionality so that the Elm maintainers can focus on awesome code-specific features such as automatic versioning and documentation generation. However, using a general-purpose code-hosting website like Github for the specific task of hosting Elm libraries means availability is dependent on Github’s availability. This is a good thing in most cases since Github has a mature infrastructure for serving code worldwide, but it also means that Elm packages are unavailable in China as collateral damage from China’s blocking of Github, and AFAIK there are no archives for Elm packages elsewhere.

For the past week and a half I’ve been staying with a friend in Shanghai. Against my better judgment, I’ve been spending some time while she’s at work programming. In particular, I’ve been working with Elm. Elm comes with a nice local package manager called elm-package which allows you to install packages listed in the official package catalog. The code for the packages themselves is hosted on the package authors’ Github accounts1, meaning each public package automatically comes with tools for filing issues, submitting patches, and reviewing source code.

Unfortunately, using elm-package behind a VPN (in my case, Astrill) is problematic. Without the VPN in China, you’re stuck, because Github is blocked. With the VPN, you may run into trouble connecting to the Elm package server, like I did. So here is a quick guide to using elm-package in China, or in general, using it with the Astrill VPN. Some of the troubleshooting will also be useful for those with bad connections (how to handle the connection dropping mid-installation, etc).

I will walk through the steps of installing elm-check. At the time of writing this guide, I am using elm-package version 0.5.1, which is part of Elm Platform 0.15.1, and Astrill version 2.9.3.

Step 1: Connecting to the package server

With Astrill running, click the SETTINGS button and enable “Smart Mode” from the drop down, as pictured:

Smart Mode dropdown

fig:The Smart Mode option

From what I’ve read, Smart Mode will only use the VPN for sites which are blocked in China, so it doesn’t affect the speed of any sites which are not blocked. Curiously, even though the Elm package server is not blocked, I have more success accessing it through the VPN with Smart Mode enabled than I do with the VPN turned off. I don’t know too much about VPNs and such so I’m not sure why this is.

Now, in the directory for your project, run elm-package install TheSeamau5/elm-check. After a few seconds, the package manager will prompt you with:

To install TheSeamau5/elm-check I would like to add the following
dependency to elm-package.json:

    "TheSeamau5/elm-check": "3.2.0 <= v < 4.0.0"

May I add that to elm-package.json for you? (y/n)

If you get this prompt, answer with a nice y. If you do not get this prompt, and instead the request times out with a failure message, try running elm-package install TheSeamau5/elm-check again. If the request takes more than 10-20 seconds and doesn’t give a timeout message, send an interrupt with ctrl-c and try again. Make sure “Smart Mode” is enabled in Astrill.

Step 2: Installing dependencies

After answering y, the next prompt should be

Some new packages are needed. Here is the upgrade plan.

  Install:
    TheSeamau5/elm-check 3.2.0
    TheSeamau5/elm-random-extra 2.1.0
    TheSeamau5/elm-shrink 2.2.1
    TheSeamau5/elm-task-extra 2.0.0
    deadfoxygrandpa/Elm-Test 1.0.4
    elm-lang/core 2.1.0
    evancz/elm-html 3.0.0
    evancz/virtual-dom 1.2.3
    maxsnew/IO 1.0.1

Do you approve of this plan? (y/n)

Once again, tell it y. This is where trouble may crop up. Once you approve the plan, the package manager will begin fetching the versioned packages from Github and installing them into the elm-stuff folder in your directory. The manager will create the directory elm-stuff/packages. For each package, you should see a message, such as

Downloading TheSeamau5/elm-check

Along with this message, the manager will create a directory

elm-stuff/packages/TheSeamau5/elm-check

but this directory will be empty until the package is downloaded. When the download is completed, the package will be added to the directory

elm-stuff/packages/TheSeamau5/elm-check/3.2.0/

and the manager will start downloading the next dependency, and you should see a message telling you that it is downloading a new package.

However, if after, say, a minute, it still just says

Downloading TheSeamau5/elm-check

with no update, your connection has probably dropped, and you should ctrl-c out, and run elm-package install, approve the plan, and let it download again. You do not need to include the package name in the command, because TheSeamau5/elm-check has already been added to elm-package.json for you. You should do the same thing if you get an error message like

Error: failed with 'Error_Packet "partial packet: expecting 1424
bytes, got: 873"' when sending request to
    <http://github.com/TheSeamau5/elm-check/zipball/3.2.0/>

or

Error: failed with 'ResponseTimeout' when sending request to
    <http://github.com/TheSeamau5/elm-check/zipball/3.2.0/>
  1. Perhaps it was just my connection and not a general VPN problem, but when I installed elm-check it hung about 3 times during the process.

With luck, you should be able to download a few packages without problem2, but if there are many dependencies to install you will probably run into the problem where installing a particular dependency hangs. But if you’ve installed, say, 4 of the 9 total dependencies before it hangs, and you re-run elm-package install, the package manager will tell you that it plans to install all 9 packages again! Instead of chancing another failure during the process of installing 9 packages, we can manually fix this:

Step 3: Manually updating the installed dependency list

The Elm package manager uses a file in elm-stuff called exact-dependencies.json to calculate the difference between requested packages in elm-package.json and your currently-installed packages. When you first run elm-package install TheSeamau5/elm-check, it adds this dependency to elm-package.json for you, but during the process of installing each dependency, it doesn’t update exact-dependencies.json. It only updates exact-dependencies.json after it has finished installing all dependencies.

This means that if the installer hangs after installing, say, 4 out of 9 dependencies, it won’t know that it has successfully installed the 4, and will try to install them again when you re-run elm-package install. We can manually create or update exact-dependencies.json so the package manager only installs what we don’t have.

Suppose we run elm-package install, and get to this point:

Downloading TheSeamau5/elm-check
Downloading TheSeamau5/elm-random-extra
Downloading TheSeamau5/elm-shrink
Downloading TheSeamau5/elm-task-extra
Downloading deadfoxygrandpa/Elm-Test
Downloading elm-lang/core

but then it hangs. This means we’ve successfully installed elm-check through Elm-Test, but the installer hung when trying to download core. We can verify that the other packages are installed by checking for them in elm-stuff/packages. Now, open elm-stuff/exact-dependencies.json, or create it if it doesn’t exist. Copy the relevant packages and their versions from the prompt about the upgrade plan we got in Step 2. In our case, this is

TheSeamau5/elm-check 3.2.0
TheSeamau5/elm-random-extra 2.1.0
TheSeamau5/elm-shrink 2.2.1
TheSeamau5/elm-task-extra 2.0.0
deadfoxygrandpa/Elm-Test 1.0.4

and add this in JSON form to exact-dependencies.json (or edit the preexisting JSON to include the new key/value mappings if the file already exists). Now, the file should look something like:

{
    "TheSeamau5/elm-check": "3.2.0",
    "TheSeamau5/elm-random-extra": "2.1.0",
    "TheSeamau5/elm-shrink": "2.2.1",
    "TheSeamau5/elm-task-extra": "2.0.0",
    "deadfoxygrandpa/Elm-Test": "1.0.4"
}

Save it, and try re-running elm-package install. Now it should prompt you with

Some new packages are needed. Here is the upgrade plan.

  Install:
    elm-lang/core 2.1.0
    evancz/elm-html 3.0.0
    evancz/virtual-dom 1.2.3
    maxsnew/IO 1.0.1

Do you approve of this plan? (y/n)

Now we only have to install the packages we don’t have yet!

Step 4: Rinse and repeat

You may have to run the previous step a few times, each time adding the packages you’ve successfully installed to exact-dependencies.json, but hopefully it will install a few more packages before failing. When the process finally completes, you will get the message

Packages configured successfully!

and you should be all good. Note that if you check exact-dependencies.json now, the package manager will have mucked with it, probably rearranging the order of the dependencies in the list, but that’s fine.

  1. There are some pros and cons to this approach, as I see it. On one hand, package names are nicely namespaced by the authors’ names, solving the problem package managers such as npm have with nice package names getting taken early. Also, as explained in the announcement post, hosting on Github means offloading much of the necessary functionality so that the Elm maintainers can focus on awesome code-specific features such as automatic versioning and documentation generation. However, using a general-purpose code-hosting website like Github for the specific task of hosting Elm libraries means availability is dependent on Github’s availability. This is a good thing in most cases since Github has a mature infrastructure for serving code worldwide, but it also means that Elm packages are unavailable in China as collateral damage from China’s blocking of Github, and AFAIK there are no archives for Elm packages elsewhere.

  2. Perhaps it was just my connection and not a general VPN problem, but when I installed elm-check it hung about 3 times during the process.

Comments