Jekyll2021-02-01T13:15:39-05:00https://briankoopman.com/Brian KoopmanInsert wit here.Brian Koopmanbrian.koopman@yale.eduProvisioning a VM for Pi-hole using Vagrant2018-01-27T16:39:00-05:002018-01-27T16:39:00-05:00https://briankoopman.com/vagrant-pihole<h1 id="testing-out-pi-hole-hold-the-pi">Testing out Pi-hole, Hold the Pi</h1>
<p>I had come across <a href="https://pi-hole.net/">Pi-hole</a> a while ago and was recently
inspired by someone on Twitter to test it out. Unfortunately I didn’t have a
Raspberry Pi on hand, having just used my Pi in a Christmas present. My
solution was to spin up a virtual machine (VM) on my desktop and test it out.
Since VirtualBox can’t emulate ARM processors we can’t actually use Raspbian,
so I used Ubuntu. The test went well and I soon had a virtualized Pi-hole
blocking ads on my network (including on my phone)!</p>
<p>When I was done for the night I shutdown the virtual machine and was sad to see
ads again, particularly on my phone. I came close to ordering another Raspberry
Pi, however decided to go ahead with a more permanent solution involving some
sort of containerization or virtualization on my home server. The server isn’t
much, just an <a href="https://ark.intel.com/products/77773/Intel-Pentium-Processor-G3220-3M-Cache-3_00-GHz">Intel Pentium G3220
Haswell</a>
with 4 GB of RAM. The limited resources made me initially think docker would be
ideal, rather than leaving a VM running on the dual-core machine all the time.</p>
<p>I don’t know much about docker, but since it was still holiday break I thought
it would be good to finally learn something about it. I ran through the <a href="https://docs.docker.com/get-started/">get
started</a> tutorial in the docker
documentation, which was actually really neat, I recommend you check it out if
you’re at all interested. However, I ended up realizing that having a web
server already running on this machine, and Pi-hole requiring use of port 80,
complicates things beyond what I wanted to deal with. I had attended a talk
about <a href="https://www.vagrantup.com/">Vagrant</a> last year, and thought I might make
use of that experience. For those who aren’t familiar with Vagrant, it is a
program which allows you to build and manage VMs, with the ability to interface
with many different services, though for this we’ll be using
<a href="https://www.virtualbox.org/">VirtualBox</a>.</p>
<p>So we’ll be walking through steps to setup vagrant on your computer, provision
a VM for use with Pi-hole, and the initial Pi-hole setup. You can run this on
any computer, though for any sort of long term use, you’ll likely want to set
this up on something stationary and always on (which I suppose is why a
Raspberry Pi makes such a great target device.)</p>
<h1 id="vagrant">Vagrant</h1>
<p>We’ll need two things,
<a href="https://wiki.archlinux.org/index.php/Vagrant">vagrant</a>, and
<a href="https://wiki.archlinux.org/index.php/VirtualBox">virtualbox</a>, which acts as a
provider for vagrant. My server runs Arch Linux, to install these we’ll just run:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">sudo </span>pacman <span class="nt">-S</span> vagrant virtualbox virtualbox-host-modules-arch</code></pre></figure>
<p>If you’re on a different system I’ll leave the installation to you.</p>
<h2 id="quick-intro">Quick Intro</h2>
<p>Vagrant has a very fast and easy to follow <a href="https://www.vagrantup.com/intro/getting-started/index.html">Getting Started
Guide</a>. The pages
are short and they go by quick. If you want to get familiar with vagrant
quickly, I’d recommend going through that. We’ll do enough to get the VM up and
running, but won’t cover everything in their guide.</p>
<p>Vagrant uses what they call a <code class="highlighter-rouge">VagrantFile</code> to configure a VM (usually referred
to as a <code class="highlighter-rouge">box</code>.) Perhaps the simplest <code class="highlighter-rouge">VagrantFile</code> defines only the VM
<code class="highlighter-rouge">box</code>. For example,</p>
<figure class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="no">Vagrant</span><span class="p">.</span><span class="nf">configure</span><span class="p">(</span><span class="s2">"2"</span><span class="p">)</span> <span class="k">do</span> <span class="o">|</span><span class="n">config</span><span class="o">|</span>
<span class="n">config</span><span class="p">.</span><span class="nf">vm</span><span class="p">.</span><span class="nf">box</span> <span class="o">=</span> <span class="s2">"ubuntu/xenial64"</span>
<span class="k">end</span></code></pre></figure>
<p>This tells vagrant that we want to use the <a href="https://app.vagrantup.com/ubuntu/boxes/xenial64">“ubuntu/xenial64” box
</a> from their publicly hosted
<a href="https://app.vagrantup.com/boxes/search">catalog of boxes</a>. Anyone can upload
these boxes, so the usual caveats about being careful with what you trust come
along with it. That said, it appears the ubuntu account looks to be official.</p>
<p>If you’d like to try this out, create a new directory and put the short
configuration above into a file called <code class="highlighter-rouge">VagrantFile</code>. Once you’ve made your
<code class="highlighter-rouge">VagrantFile</code> all that’s needed is to run:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">vagrant up</code></pre></figure>
<p>This will pull the xenial64 box and start it up. You can open VirtualBox and
watch this happen. You can then connect to the new machine with:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">vagrant ssh</code></pre></figure>
<p>You should be at a prompt within the VM now. Neat, right?</p>
<h2 id="provisioning-the-pi-hole-vm">Provisioning the Pi-hole VM</h2>
<p>We’ll make just a few changes to that basic VagrantFile.</p>
<figure class="highlight"><pre><code class="language-ruby" data-lang="ruby"><table class="rouge-table"><tbody><tr><td class="gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
</pre></td><td class="code"><pre><span class="c1"># -*- mode: ruby -*-</span>
<span class="c1"># vi: set ft=ruby :</span>
<span class="no">Vagrant</span><span class="p">.</span><span class="nf">configure</span><span class="p">(</span><span class="s2">"2"</span><span class="p">)</span> <span class="k">do</span> <span class="o">|</span><span class="n">config</span><span class="o">|</span>
<span class="n">config</span><span class="p">.</span><span class="nf">vm</span><span class="p">.</span><span class="nf">box</span> <span class="o">=</span> <span class="s2">"ubuntu/xenial64"</span>
<span class="n">config</span><span class="p">.</span><span class="nf">vm</span><span class="p">.</span><span class="nf">hostname</span> <span class="o">=</span> <span class="s2">"pihole"</span>
<span class="n">config</span><span class="p">.</span><span class="nf">vm</span><span class="p">.</span><span class="nf">provider</span> <span class="s2">"virtualbox"</span> <span class="k">do</span> <span class="o">|</span><span class="n">vb</span><span class="o">|</span>
<span class="n">vb</span><span class="p">.</span><span class="nf">name</span> <span class="o">=</span> <span class="s2">"pihole"</span>
<span class="n">vb</span><span class="p">.</span><span class="nf">customize</span> <span class="p">[</span><span class="s2">"modifyvm"</span><span class="p">,</span> <span class="ss">:id</span><span class="p">,</span> <span class="s2">"--cpus"</span><span class="p">,</span> <span class="s2">"1"</span><span class="p">]</span>
<span class="n">vb</span><span class="p">.</span><span class="nf">customize</span> <span class="p">[</span><span class="s2">"modifyvm"</span><span class="p">,</span> <span class="ss">:id</span><span class="p">,</span> <span class="s2">"--memory"</span><span class="p">,</span> <span class="s2">"1024"</span><span class="p">]</span>
<span class="n">vb</span><span class="p">.</span><span class="nf">customize</span> <span class="p">[</span><span class="s2">"modifyvm"</span><span class="p">,</span> <span class="ss">:id</span><span class="p">,</span> <span class="s2">"--usb"</span><span class="p">,</span> <span class="s2">"off"</span><span class="p">]</span>
<span class="n">vb</span><span class="p">.</span><span class="nf">customize</span> <span class="p">[</span><span class="s2">"modifyvm"</span><span class="p">,</span> <span class="ss">:id</span><span class="p">,</span> <span class="s2">"--usbehci"</span><span class="p">,</span> <span class="s2">"off"</span><span class="p">]</span>
<span class="k">end</span>
<span class="n">config</span><span class="p">.</span><span class="nf">vm</span><span class="p">.</span><span class="nf">network</span> <span class="s2">"public_network"</span><span class="p">,</span> <span class="ss">ip: </span><span class="s2">"192.168.1.100"</span>
<span class="k">end</span></pre></td></tr></tbody></table></code></pre></figure>
<p>Let’s walk through this and see what’s going on, starting with lines 1 and 2:</p>
<figure class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="c1"># -*- mode: ruby -*-</span>
<span class="c1"># vi: set ft=ruby :</span></code></pre></figure>
<p>These are modelines, telling
<a href="https://www.gnu.org/software/emacs/manual/html_node/emacs/Choosing-Modes.html">emacs</a>
and <a href="http://vim.wikia.com/wiki/Modeline_magic">vi</a> that we’re looking at ruby,
which is what the VagrantFiles are in. These aren’t required, but they’re common to see.</p>
<p>Line 4, the <code class="highlighter-rouge">Vagrant.configure("2")</code> line, tells Vagrant which <a href="https://www.vagrantup.com/docs/vagrantfile/version.html">configuration
version</a> we’re using.
Don’t change this. We already know <code class="highlighter-rouge">config.vm.box</code> specifies the box we want to
pull. <code class="highlighter-rouge">config.vm.hostname</code> sets the hostname we’ll use on the VM to “pihole”.
<code class="highlighter-rouge">config.vm.provider</code> sets the provider to be “virtualbox”, which is is by
default, but this allows us to specify some more options (which we’ll run
through in a minute. Finally, <code class="highlighter-rouge">config.vm.network</code> specifies the network to be a
“public_network”, which for VirtualBox makes it a “Bridged Adapter”, this
allows us to assign the VM an IP as if it is directly connected to our local
network. We then set this IP on the rest of the line to be <code class="highlighter-rouge">192.168.1.100</code>. Set
this to whatever you want, ideally outside of your router’s DHCP address range.</p>
<p>Let’s look now at the options set in <code class="highlighter-rouge">config.vm.provider</code>, lines 7-13.</p>
<figure class="highlight"><pre><code class="language-ruby" data-lang="ruby"> <span class="n">config</span><span class="p">.</span><span class="nf">vm</span><span class="p">.</span><span class="nf">provider</span> <span class="s2">"virtualbox"</span> <span class="k">do</span> <span class="o">|</span><span class="n">vb</span><span class="o">|</span>
<span class="n">vb</span><span class="p">.</span><span class="nf">name</span> <span class="o">=</span> <span class="s2">"pihole"</span>
<span class="n">vb</span><span class="p">.</span><span class="nf">customize</span> <span class="p">[</span><span class="s2">"modifyvm"</span><span class="p">,</span> <span class="ss">:id</span><span class="p">,</span> <span class="s2">"--cpus"</span><span class="p">,</span> <span class="s2">"1"</span><span class="p">]</span>
<span class="n">vb</span><span class="p">.</span><span class="nf">customize</span> <span class="p">[</span><span class="s2">"modifyvm"</span><span class="p">,</span> <span class="ss">:id</span><span class="p">,</span> <span class="s2">"--memory"</span><span class="p">,</span> <span class="s2">"1024"</span><span class="p">]</span>
<span class="n">vb</span><span class="p">.</span><span class="nf">customize</span> <span class="p">[</span><span class="s2">"modifyvm"</span><span class="p">,</span> <span class="ss">:id</span><span class="p">,</span> <span class="s2">"--usb"</span><span class="p">,</span> <span class="s2">"off"</span><span class="p">]</span>
<span class="n">vb</span><span class="p">.</span><span class="nf">customize</span> <span class="p">[</span><span class="s2">"modifyvm"</span><span class="p">,</span> <span class="ss">:id</span><span class="p">,</span> <span class="s2">"--usbehci"</span><span class="p">,</span> <span class="s2">"off"</span><span class="p">]</span>
<span class="k">end</span></code></pre></figure>
<p>This does the resource limiting for the VM, limiting the box to use of 1 CPU,
and 1024 MB of RAM. (Note with this few resources, it could take a little while
to boot, be patient. Vagrant defaults to a 5 minute timeout, if it doesn’t boot
by then it’ll throw a warning. If you can, you can try increasing the resources
available, or increase the default timeout with <code class="highlighter-rouge">config.vm.boot_timeout</code>
provided in seconds.) The usb options explicitly disable USB passthrough. I do
this simply to ensure any USB device I plug into the server doesn’t show up on
the VM too.</p>
<p>Then to provision the box we just need to run <code class="highlighter-rouge">vagrant up</code>.</p>
<p>A successful <code class="highlighter-rouge">vagrant up</code> will produce output similar to this:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">[</span>koopman@honey pihole]<span class="nv">$ </span>vagrant up
Bringing machine <span class="s1">'default'</span> up with <span class="s1">'virtualbox'</span> provider...
<span class="o">==></span> default: Importing base box <span class="s1">'ubuntu/xenial64'</span>...
<span class="o">==></span> default: Matching MAC address <span class="k">for </span>NAT networking...
<span class="o">==></span> default: Checking <span class="k">if </span>box <span class="s1">'ubuntu/xenial64'</span> is up to date...
<span class="o">==></span> default: Setting the name of the VM: pihole
<span class="o">==></span> default: Clearing any previously <span class="nb">set </span>network interfaces...
<span class="o">==></span> default: Preparing network interfaces based on configuration...
default: Adapter 1: nat
default: Adapter 2: bridged
<span class="o">==></span> default: Forwarding ports...
default: 22 <span class="o">(</span>guest<span class="o">)</span> <span class="o">=></span> 2222 <span class="o">(</span>host<span class="o">)</span> <span class="o">(</span>adapter 1<span class="o">)</span>
<span class="o">==></span> default: Running <span class="s1">'pre-boot'</span> VM customizations...
<span class="o">==></span> default: Booting VM...
<span class="o">==></span> default: Waiting <span class="k">for </span>machine to boot. This may take a few minutes...
default: SSH address: 127.0.0.1:2222
default: SSH username: ubuntu
default: SSH auth method: password
default: Warning: Connection reset. Retrying...
default: Warning: Remote connection disconnect. Retrying...
default: Warning: Connection reset. Retrying...
default: Warning: Remote connection disconnect. Retrying...
default: Warning: Connection reset. Retrying...
default: Warning: Remote connection disconnect. Retrying...
default: Warning: Connection reset. Retrying...
default: Warning: Remote connection disconnect. Retrying...
default: Warning: Connection reset. Retrying...
default: Warning: Remote connection disconnect. Retrying...
default: Warning: Connection reset. Retrying...
default: Warning: Remote connection disconnect. Retrying...
default: Warning: Connection reset. Retrying...
default: Warning: Remote connection disconnect. Retrying...
default: Warning: Connection reset. Retrying...
default: Warning: Remote connection disconnect. Retrying...
default:
default: Inserting generated public key within guest...
default: Removing insecure key from the guest <span class="k">if </span>it<span class="s1">'s present...
default: Key inserted! Disconnecting and reconnecting using new SSH key...
==> default: Machine booted and ready!
==> default: Checking for guest additions in VM...
default: The guest additions on this VM do not match the installed version of
default: VirtualBox! In most cases this is fine, but in rare cases it can
default: prevent things such as shared folders from working properly. If you see
default: shared folder errors, please make sure the guest additions within the
default: virtual machine match the version of VirtualBox you have installed on
default: your host and reload your VM.
default:
default: Guest Additions Version: 5.0.40
default: VirtualBox Version: 5.2
==> default: Setting hostname...
==> default: Configuring and enabling network interfaces...
==> default: Mounting shared folders...
default: /vagrant => /home/koopman/vagrant/pihole</span></code></pre></figure>
<p>If this is your first time running <code class="highlighter-rouge">vagrant up</code> it’ll also spend time
downloading the box image. Note the <code class="highlighter-rouge">Connection reset</code> and <code class="highlighter-rouge">Remote connection
disconnect</code> warnings are just us trying to ssh before the ssh service has come
online. You can now connect to the VM with <code class="highlighter-rouge">vagrant ssh</code>.</p>
<h1 id="installing-pi-hole">Installing Pi-hole</h1>
<p>Now we just need to install Pi-hole. For this I refer to their <a href="https://github.com/pi-hole/pi-hole#one-step-automated-install">github
page</a>. They
provide a simple one-liner which pipes to bash, not ideal. During my first
installation I followed “Alternative Method 2” and checked out the script I was
running. If you’re trusting enough, use the one-liner. We have to check one
thing before moving forward though, our static IP interface, to do so run
<code class="highlighter-rouge">ifconfig</code>, and note which interface is associated with your assigned static IP
(in this example <code class="highlighter-rouge">192.168.1.100</code>), for instance <code class="highlighter-rouge">enp0s8</code>. Once you have noted
that, run installer.</p>
<p>I won’t cover most of the dialogues, they’re pretty straight forward. You’ll want
to select the interface we noted above at the “Chose An Interface” option. (If
you forgot to note which interface, I can almost guarantee it’ll be the second
option, since vagrant sets up two network interfaces, the first for its own
use, the other for the public network.</p>
<p>Pi-hole needs to know where to point its upstream DNS queries, since it acts
as a middle man. Use whatever you feel like here, Google is a safe bet. I
decided to try out Quad9. This is also configurable from the Pihole interface
so don’t spend too much time thinking here.</p>
<p>The potentially tricky part is the network configuration, you’ll reach a screen
that looks like this:</p>
<p><img src="https://briankoopman.com/assets/images/pihole_setup/step0.png" alt="step0" /></p>
<p>We don’t want to use this, as it is for the first network interface, not our
public network bridged adapter, so select “No”. Then we’ll enter our assigned
IP from the VagrantFile like so:</p>
<p><img src="https://briankoopman.com/assets/images/pihole_setup/step1.png" alt="step1" /></p>
<p>This is followed by specifying our gateway, which is likely your router,
commonly (and in my case) <code class="highlighter-rouge">192.168.1.1</code>, so enter that here:</p>
<p><img src="https://briankoopman.com/assets/images/pihole_setup/step2.png" alt="step2" /></p>
<p>You’ll then be asked to verify the settings, it should look something like
this:</p>
<p><img src="https://briankoopman.com/assets/images/pihole_setup/step3.png" alt="step3" /></p>
<p>The last two prompts are about the admin interface, which is neat, you should
install it.</p>
<p>If everything goes smoothly you’ll land on an “Installation Complete!” page
like this:</p>
<p><img src="https://briankoopman.com/assets/images/pihole_setup/step4.png" alt="step4" /></p>
<p>You should note down the Admin login info, as it can’t be retrieved after this.
(Note if you need to redo this for whatever reason, just run the installer
again and select reconfigure. In more dire circumstances we can remove the VM
and reprovision with <code class="highlighter-rouge">vagrant destroy</code> and another <code class="highlighter-rouge">vagrant up</code>.)</p>
<p>Now we can point our browser to <code class="highlighter-rouge">192.168.1.100/admin</code>, you should see the
pi-hole dashboard:</p>
<p><img src="https://briankoopman.com/assets/images/pihole_setup/dashboard.png" alt="dashboard" /></p>
<h1 id="configure-your-devices">Configure Your Devices</h1>
<p>You have several options for configuring your devices to use your new VM as a
DNS provider. They are well enumerated on the <a href="https://discourse.pi-hole.net/t/how-do-i-configure-my-devices-to-use-pi-hole-as-their-dns-server/245">Pi-hole
discourse</a>.</p>
<p>I would prefer to go with option 1, configuring the Pi-hole as my router’s only
DNS provider, however I don’t want this experiment to negatively affect my
housemates, so I went with option 3, configuring each device individually. This
allows me to use the Pi-hole and doesn’t force it upon my housemates. I have
several devices with different configurations, so I’ll go over configuring each
of them.</p>
<h2 id="android">Android</h2>
<p>To start, my phone, perhaps the primary motivation for setting this up, since
Google has been forcing links to open in Chrome, which lacks ad block on
mobile. I use an Android phone, which at the time of this write up is running
version 8.0.0.</p>
<p>To setup the pi-hole as your Android phone’s DNS provider navigate to the Wi-Fi
settings and click and hold on your home network, this should pop up a menu as
shown here:</p>
<p><a href="/assets/images/pihole_setup/android_step0.png"><img src="/assets/images/pihole_setup/android_step0.png" style="width: 400px;" /></a></p>
<p>Click on “Modify network”. Scroll down to “IP settings”, where we want to
select “Static”. Unless you’ve configured this previously it is likely on
“DHCP”.</p>
<p><a href="/assets/images/pihole_setup/android_step1.png"><img src="/assets/images/pihole_setup/android_step1.png" style="width: 400px;" /></a></p>
<p>This will allow for input of a static IP address for your phone, along with all
the other necessary details, including of course the DNS provider. Here we’ll
enter the IP of our Pi-hole, <code class="highlighter-rouge">192.168.1.100</code> as shown here:</p>
<p><a href="/assets/images/pihole_setup/android_step2.png"><img src="/assets/images/pihole_setup/android_step2.png" style="width: 400px;" /></a></p>
<p>I find I then have to toggle Wi-Fi off and on again in order to fully
reconnect, else it might look like you’re connected, but you won’t actually
have Internet access.</p>
<h2 id="linux">Linux</h2>
<p>I am running Arch Linux on my desktop and laptop. On the desktop I am using
dhcpcd and on the laptop NetworkManager (with nm-applet). Their configuration
differs slightly.</p>
<h3 id="dhcpcd">dhcpcd</h3>
<p>I should have setup a static IP long ago for the desktop, to do so, add these
details to your <code class="highlighter-rouge">/etc/dhcpcd.conf</code> file:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">interface eth0
static <span class="nv">ip_address</span><span class="o">=</span>192.168.1.64/24
static <span class="nv">routers</span><span class="o">=</span>192.168.1.1
static <span class="nv">domain_name_servers</span><span class="o">=</span>192.168.1.100</code></pre></figure>
<p>Be sure to use an available IP address, and again one outside of your router’s
DHCP range.</p>
<h3 id="nm-applet">nm-applet</h3>
<p><code class="highlighter-rouge">nm-applet</code> is similarly easy to configure, just open up the the connection
editor, navigate over the IPv4 Settings and enter the following, substituting
in a suitable IP address:</p>
<p><img src="https://briankoopman.com/assets/images/pihole_setup/nm_applet_config.png" alt="nm_applet_config" /></p>
<h1 id="wrapping-up">Wrapping Up</h1>
<p>That’s it! We’ve created a Pi-hole instance on a virtualized Ubuntu box using
vagrant and configured our devices to use it as a DNS provider. You can use the
Pi-hole dashboard to view how many blocked DNS requests you make on which
devices.</p>
<p>If you ever want to undo all this though, perhaps so you can move to a
Raspberry Pi, you will have to revert the individual devices’ network
configurations. You can then bring down the VM by navigating to the same
directory where your <code class="highlighter-rouge">VagrantFile</code> lives and run <code class="highlighter-rouge">vagrant destroy</code>. It’ll
prompt you to make sure you want to actually do it, but then the VM will be
gone.</p>Brian Koopmanbrian.koopman@yale.eduA quick guide to setting up Pi-hole on a virtual machine provisioned with vagrant.Photo Gallery - Japan Trip2017-09-16T20:19:00-04:002017-09-16T20:19:00-04:00https://briankoopman.com/photo-gallery-japan-trip<p>This is just a dump of a ton of photos from my trip to Japan. Hopefully will be
a bit more organized/descriptive in the future.</p>
<h1 id="kyoto">Kyoto</h1>
<figure class="third full">
<a href="/assets/images/japan/kyoto/IMG_20170722_092348_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_092348_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_103738_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_103738_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_143558_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_143558_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_153244_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_153244_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_154451_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_154451_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_154746_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_154746_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_154803_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_154803_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_160151_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_160151_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_160233_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_160233_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_160254_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_160254_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_160319_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_160319_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_160324_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_160324_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_160335_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_160335_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_160348_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_160348_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_160422_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_160422_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_160609_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_160609_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_160631_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_160631_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_160633_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_160633_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_160750_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_160750_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_160818_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_160818_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_161046_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_161046_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_161110_1_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_161110_1_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_161520_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_161520_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_161850_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_161850_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_161915_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_161915_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_162139_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_162139_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_162152_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_162152_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_162157_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_162157_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_162217_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_162217_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_162231_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_162231_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_162254_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_162254_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_162306_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_162306_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_162318_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_162318_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_162609_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_162609_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_162708_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_162708_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_162720_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_162720_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_162756_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_162756_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_162943_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_162943_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_162954_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_162954_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_163142_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_163142_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_165829_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_165829_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_170355_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_170355_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_170728_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_170728_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_171543_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_171543_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_172001_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_172001_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_172024_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_172024_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_172059_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_172059_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_172525_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_172525_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_172528_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_172528_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_190040_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_190040_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_190923_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_190923_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_194058_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_194058_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_194455_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_194455_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_204749_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_204749_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_204752_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_204752_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_210943_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_210943_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170722_211134_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170722_211134_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_102144_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_102144_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_102247_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_102247_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_102812_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_102812_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_102824_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_102824_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_103111_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_103111_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_103638_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_103638_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_103703_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_103703_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_103759_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_103759_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_104122_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_104122_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_121016_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_121016_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_122229_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_122229_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_122517_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_122517_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_122842_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_122842_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_122924_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_122924_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_123120_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_123120_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_123123_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_123123_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_123201_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_123201_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_123246_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_123246_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_123256_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_123256_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_123305_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_123305_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_123318_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_123318_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_123341_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_123341_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_123437_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_123437_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_123507_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_123507_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_123517_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_123517_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_123547_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_123547_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_123604_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_123604_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_123629_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_123629_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_123724_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_123724_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_123729_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_123729_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_123915_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_123915_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_124031_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_124031_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_124126_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_124126_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_124133_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_124133_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_124141_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_124141_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_124205_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_124205_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_124250_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_124250_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_124428_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_124428_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_124437_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_124437_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_124442_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_124442_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_124606_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_124606_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_124612_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_124612_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_124630_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_124630_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_124654_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_124654_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_124711_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_124711_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_124723_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_124723_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_124901_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_124901_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_125035_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_125035_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_125853_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_125853_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_125915_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_125915_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_125958_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_125958_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_130037_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_130037_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_130630_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_130630_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_131846_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_131846_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_132140_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_132140_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_132310_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_132310_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_132426_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_132426_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_132434_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_132434_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_132437_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_132437_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_132932_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_132932_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_133234_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_133234_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_133329_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_133329_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_142056_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_142056_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_142107_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_142107_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_142436_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_142436_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_142611_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_142611_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_142640_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_142640_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_142717_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_142717_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_142836_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_142836_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_142845_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_142845_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_143105_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_143105_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_143144_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_143144_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_143148_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_143148_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_143155_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_143155_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_143211_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_143211_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_143422_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_143422_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_143430_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_143430_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_144024_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_144024_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_144026_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_144026_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_144241_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_144241_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_144432_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_144432_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_144444_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_144444_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_144510_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_144510_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_145109_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_145109_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_145120_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_145120_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_145312_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_145312_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_183823_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_183823_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_184955_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_184955_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170723_220232_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170723_220232_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170724_100343_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170724_100343_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170724_100643_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170724_100643_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170724_103925_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170724_103925_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170724_104303_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170724_104303_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170724_104455_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170724_104455_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170724_104553_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170724_104553_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170724_105137_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170724_105137_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170724_105141_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170724_105141_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170724_110230_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170724_110230_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170724_111631_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170724_111631_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170724_112107_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170724_112107_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170724_115104_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170724_115104_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170724_115212_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170724_115212_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170724_115418_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170724_115418_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170724_115500_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170724_115500_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170724_115717_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170724_115717_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170724_120745_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170724_120745_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170724_124456_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170724_124456_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/kyoto/IMG_20170724_131006_scaled.jpg">
<img src="/assets/images/japan/kyoto/IMG_20170724_131006_scaled.jpg" alt="" />
</a>
<figcaption>Photos from Kyoto.
</figcaption>
</figure>
<h1 id="osaka">Osaka</h1>
<figure class="third full">
<a href="/assets/images/japan/osaka/IMG_20170724_134425_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_134425_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_134852_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_134852_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_134900_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_134900_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_140209_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_140209_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_145214_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_145214_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_145222_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_145222_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_154156_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_154156_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_154502_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_154502_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_154601_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_154601_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_154732_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_154732_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_154759_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_154759_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_154832_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_154832_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_154835_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_154835_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_154839_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_154839_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_154842_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_154842_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_154951_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_154951_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_154954_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_154954_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_155026_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_155026_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_155034_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_155034_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_155148_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_155148_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_155217_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_155217_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_155223_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_155223_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_155405_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_155405_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_155429_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_155429_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_155557_1_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_155557_1_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_162458_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_162458_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_162501_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_162501_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_162505_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_162505_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_162557_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_162557_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_162640_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_162640_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_162644_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_162644_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_162646_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_162646_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_162809_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_162809_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_162855_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_162855_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_162940_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_162940_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_162943_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_162943_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_162953_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_162953_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_163109_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_163109_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_163141_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_163141_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_165607_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_165607_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_165641_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_165641_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_170214_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_170214_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_170310_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_170310_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_170345_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_170345_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_170438_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_170438_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_171051_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_171051_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_171620_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_171620_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_171744_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_171744_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_171938_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_171938_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_172658_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_172658_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_173027_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_173027_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_192537_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_192537_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_195604_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_195604_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_202605_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_202605_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_202832_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_202832_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_203316_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_203316_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_203321_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_203321_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_203533_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_203533_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_203538_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_203538_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_203543_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_203543_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_204637_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_204637_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_204641_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_204641_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_204645_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_204645_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_204649_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_204649_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_204801_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_204801_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_204934_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_204934_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_221611_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_221611_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_221858_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_221858_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/IMG_20170724_221948_scaled.jpg">
<img src="/assets/images/japan/osaka/IMG_20170724_221948_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/osaka/PANO_20170724_162509_scaled.jpg">
<img src="/assets/images/japan/osaka/PANO_20170724_162509_scaled.jpg" alt="" />
</a>
<figcaption>Photos from Osaka.
</figcaption>
</figure>
<h1 id="hakone">Hakone</h1>
<figure class="third full">
<a href="/assets/images/japan/hakone/IMG_20170725_115628_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170725_115628_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170725_115845_1_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170725_115845_1_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170725_115845_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170725_115845_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170725_120553_1_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170725_120553_1_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170725_120553_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170725_120553_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170725_120554_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170725_120554_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170725_120555_1_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170725_120555_1_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170725_120555_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170725_120555_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170725_121756_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170725_121756_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170725_122108_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170725_122108_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170725_122410_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170725_122410_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170725_122411_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170725_122411_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170725_122416_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170725_122416_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170725_124855_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170725_124855_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170725_130230_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170725_130230_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170725_130509_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170725_130509_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170725_130511_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170725_130511_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170725_130512_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170725_130512_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170725_140305_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170725_140305_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170725_141808_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170725_141808_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170725_141809_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170725_141809_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170725_155709_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170725_155709_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170725_160056_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170725_160056_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170725_160100_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170725_160100_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_074227_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_074227_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_074235_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_074235_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_094201_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_094201_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_094210_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_094210_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_094345_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_094345_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_095012_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_095012_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_101916_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_101916_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_101928_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_101928_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_102119_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_102119_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_102213_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_102213_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_102232_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_102232_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_102317_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_102317_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_102338_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_102338_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_103040_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_103040_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_103111_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_103111_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_103604_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_103604_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_103636_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_103636_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_103655_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_103655_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_103658_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_103658_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_103739_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_103739_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_103846_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_103846_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_104201_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_104201_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_104929_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_104929_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_105036_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_105036_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_105041_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_105041_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_110508_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_110508_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_111903_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_111903_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_112816_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_112816_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_121009_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_121009_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_124142_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_124142_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_124735_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_124735_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_124856_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_124856_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_124900_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_124900_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_130409_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_130409_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_130417_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_130417_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_131845_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_131845_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_132235_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_132235_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_132429_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_132429_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_132437_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_132437_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_132527_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_132527_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_133035_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_133035_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_133218_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_133218_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_133619_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_133619_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_133628_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_133628_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_133734_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_133734_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_133739_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_133739_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_133750_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_133750_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_133955_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_133955_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_134125_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_134125_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_134246_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_134246_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_134404_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_134404_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/IMG_20170726_135312_scaled.jpg">
<img src="/assets/images/japan/hakone/IMG_20170726_135312_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hakone/PANO_20170726_105048_scaled.jpg">
<img src="/assets/images/japan/hakone/PANO_20170726_105048_scaled.jpg" alt="" />
</a>
<figcaption>Photos from Hakone.
</figcaption>
</figure>
<h1 id="tokyo">Tokyo</h1>
<figure class="third full">
<a href="/assets/images/japan/tokyo/IMG_20170726_204929_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170726_204929_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170726_205708_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170726_205708_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170726_221017_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170726_221017_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_104337_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_104337_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_104457_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_104457_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_105456_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_105456_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_105501_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_105501_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_105847_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_105847_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_105910_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_105910_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_110600_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_110600_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_110819_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_110819_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_110913_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_110913_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_111011_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_111011_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_111828_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_111828_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_112949_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_112949_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_112956_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_112956_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_112959_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_112959_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_113229_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_113229_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_113251_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_113251_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_113514_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_113514_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_113518_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_113518_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_113755_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_113755_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_113835_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_113835_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_115551_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_115551_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_132437_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_132437_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_132456_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_132456_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_132624_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_132624_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_132650_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_132650_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_132830_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_132830_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_133123_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_133123_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_133242_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_133242_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_134858_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_134858_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_142059_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_142059_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_142109_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_142109_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_143309_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_143309_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_143316_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_143316_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_143617_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_143617_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_143920_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_143920_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_144424_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_144424_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_144642_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_144642_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_144757_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_144757_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_145317_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_145317_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_145548_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_145548_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_150637_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_150637_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_170122_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_170122_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_184617_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_184617_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_201141_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_201141_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_203139_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_203139_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_203532_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_203532_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_203537_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_203537_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_203540_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_203540_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_203549_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_203549_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_203621_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_203621_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_203623_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_203623_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_204313_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_204313_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_204538_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_204538_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_204545_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_204545_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_211737_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_211737_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_212120_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_212120_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_212132_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_212132_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_212234_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_212234_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_212557_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_212557_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_212647_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_212647_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_212657_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_212657_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_212855_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_212855_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_213231_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_213231_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_214134_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_214134_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_214153_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_214153_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_214449_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_214449_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_215617_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_215617_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_220005_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_220005_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/tokyo/IMG_20170727_220705_scaled.jpg">
<img src="/assets/images/japan/tokyo/IMG_20170727_220705_scaled.jpg" alt="" />
</a>
<figcaption>Photos from Tokyo.
</figcaption>
</figure>
<h1 id="hiroshima">Hiroshima</h1>
<figure class="third full">
<a href="/assets/images/japan/hiroshima/IMG_20170728_150842_scaled.jpg">
<img src="/assets/images/japan/hiroshima/IMG_20170728_150842_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hiroshima/IMG_20170728_154638_scaled.jpg">
<img src="/assets/images/japan/hiroshima/IMG_20170728_154638_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hiroshima/IMG_20170728_155114_scaled.jpg">
<img src="/assets/images/japan/hiroshima/IMG_20170728_155114_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hiroshima/IMG_20170728_155400_scaled.jpg">
<img src="/assets/images/japan/hiroshima/IMG_20170728_155400_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hiroshima/IMG_20170728_155639_scaled.jpg">
<img src="/assets/images/japan/hiroshima/IMG_20170728_155639_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hiroshima/IMG_20170728_155952_scaled.jpg">
<img src="/assets/images/japan/hiroshima/IMG_20170728_155952_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hiroshima/IMG_20170728_160003_scaled.jpg">
<img src="/assets/images/japan/hiroshima/IMG_20170728_160003_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hiroshima/IMG_20170728_160038_scaled.jpg">
<img src="/assets/images/japan/hiroshima/IMG_20170728_160038_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hiroshima/IMG_20170728_160149_scaled.jpg">
<img src="/assets/images/japan/hiroshima/IMG_20170728_160149_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hiroshima/IMG_20170728_160153_scaled.jpg">
<img src="/assets/images/japan/hiroshima/IMG_20170728_160153_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hiroshima/IMG_20170728_160158_scaled.jpg">
<img src="/assets/images/japan/hiroshima/IMG_20170728_160158_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hiroshima/IMG_20170728_160419_scaled.jpg">
<img src="/assets/images/japan/hiroshima/IMG_20170728_160419_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hiroshima/IMG_20170728_160608_scaled.jpg">
<img src="/assets/images/japan/hiroshima/IMG_20170728_160608_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hiroshima/IMG_20170728_160648_scaled.jpg">
<img src="/assets/images/japan/hiroshima/IMG_20170728_160648_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hiroshima/IMG_20170728_160747_scaled.jpg">
<img src="/assets/images/japan/hiroshima/IMG_20170728_160747_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hiroshima/IMG_20170728_161114_scaled.jpg">
<img src="/assets/images/japan/hiroshima/IMG_20170728_161114_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hiroshima/IMG_20170728_161323_scaled.jpg">
<img src="/assets/images/japan/hiroshima/IMG_20170728_161323_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hiroshima/IMG_20170728_165457_1_scaled.jpg">
<img src="/assets/images/japan/hiroshima/IMG_20170728_165457_1_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hiroshima/IMG_20170728_170409_scaled.jpg">
<img src="/assets/images/japan/hiroshima/IMG_20170728_170409_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hiroshima/IMG_20170728_170533_scaled.jpg">
<img src="/assets/images/japan/hiroshima/IMG_20170728_170533_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hiroshima/IMG_20170728_170611_scaled.jpg">
<img src="/assets/images/japan/hiroshima/IMG_20170728_170611_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hiroshima/IMG_20170728_170654_scaled.jpg">
<img src="/assets/images/japan/hiroshima/IMG_20170728_170654_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hiroshima/IMG_20170728_170709_scaled.jpg">
<img src="/assets/images/japan/hiroshima/IMG_20170728_170709_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hiroshima/IMG_20170728_170911_scaled.jpg">
<img src="/assets/images/japan/hiroshima/IMG_20170728_170911_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hiroshima/IMG_20170728_171335_scaled.jpg">
<img src="/assets/images/japan/hiroshima/IMG_20170728_171335_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hiroshima/IMG_20170728_172210_scaled.jpg">
<img src="/assets/images/japan/hiroshima/IMG_20170728_172210_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hiroshima/IMG_20170728_172220_scaled.jpg">
<img src="/assets/images/japan/hiroshima/IMG_20170728_172220_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hiroshima/IMG_20170728_172244_scaled.jpg">
<img src="/assets/images/japan/hiroshima/IMG_20170728_172244_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hiroshima/IMG_20170728_172253_scaled.jpg">
<img src="/assets/images/japan/hiroshima/IMG_20170728_172253_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hiroshima/IMG_20170728_172322_scaled.jpg">
<img src="/assets/images/japan/hiroshima/IMG_20170728_172322_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hiroshima/IMG_20170728_172325_scaled.jpg">
<img src="/assets/images/japan/hiroshima/IMG_20170728_172325_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hiroshima/IMG_20170728_172332_scaled.jpg">
<img src="/assets/images/japan/hiroshima/IMG_20170728_172332_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hiroshima/IMG_20170728_172615_scaled.jpg">
<img src="/assets/images/japan/hiroshima/IMG_20170728_172615_scaled.jpg" alt="" />
</a>
<a href="/assets/images/japan/hiroshima/IMG_20170728_172635_scaled.jpg">
<img src="/assets/images/japan/hiroshima/IMG_20170728_172635_scaled.jpg" alt="" />
</a>
<figcaption>Photos from Hiroshima.
</figcaption>
</figure>Brian Koopmanbrian.koopman@yale.eduRather large, somewhat unorganized, photo gallery from my trip to Japan in July.New Jekyll Powered Website2017-02-14T23:50:00-05:002017-02-14T23:50:00-05:00https://briankoopman.com/new-jekyll-powered-site<p>This shiny new site layout is thanks to a recent move to using
<a href="https://jekyllrb.com/">Jekyll</a> with the amazingly good looking <a href="https://mmistakes.github.io/minimal-mistakes/">Minimal
Mistakes</a> theme by Michael
Rose. The move was partly inspired by seeing a friend switch his site over a
while back (<a href="https://duetosymmetry.com/">duetosymmetry</a>), and partly motivated
by not loving my previous platform, Ghost, as much as I did when I originally
found it.</p>
<p>Just spending the past few days doing some minor formatting tweaks to the posts
I migrated from Ghost make me wonder why I didn’t switch long ago. I won’t go
into details now, perhaps I’ll write up why (and how) I migrated with some
initial setup tips for Jekyll. For now I just wanted to announce the move, if
it wasn’t obvious. Some content has moved/been taken down. Expect things to
shuffle/come back online (like comments) as I work out the final kinks.</p>Brian Koopmanbrian.koopman@yale.eduNew site layout made with Jekyll. We're expecting some minor turbulance, please fasten your seat belts.ZFS: Automated Snapshots2015-06-29T00:47:26-04:002015-06-29T00:47:26-04:00https://briankoopman.com/zfs-automated-snapshots<p>Snapshots allow you to copy the state of your zpool at a given instance. This
can be useful in many ways. For instance, if you delete a file that you still
want and you have a snapshot of a time when you had the file you can go
retrieve it. Snapshots are cheap to make, so one is encouraged to make them
often. Before setting up automated snapshots I had only ever made two
snapshots. You can view your snapshots with <code class="highlighter-rouge">zfs list -t snapshot</code>:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">sudo </span>zfs list <span class="nt">-t</span> snapshot
<span class="o">[</span><span class="nb">sudo</span><span class="o">]</span> password <span class="k">for </span>koopman:
NAME USED AVAIL REFER MOUNTPOINT
lotus/koopman@fresh 84K - 112K -
lotus/koopman@home_restored 2.36G - 138G -</code></pre></figure>
<p>In this output <code class="highlighter-rouge">fresh</code> was made when I had just made the zpool and
<code class="highlighter-rouge">home_restored</code> was made just after I had restored my home directory from
backup to the zpool.</p>
<p>This is a great start, but we don’t want to just rely on making snapshots by
hand. It would be more useful if we have some automated system for taking
snapshots. Thankfully, such tools already exists. I am going to install
<a href="https://aur.archlinux.org/packages/zfs-auto-snapshot-git/">zfs-auto-snapshot-git</a>
from the AUR, which is from the folks at zfsonlinux. Upon installation we will
see the following output:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">==</span> Enable and start snapshot timers as needed:
<span class="o">==</span> systemctl <span class="nb">enable </span>zfs-auto-snapshot-daily.timer
<span class="o">==</span> available intervals: frequent, hourly, daily, weekly, monthly
<span class="o">==</span> Recursively <span class="o">(</span><span class="nt">-r</span><span class="o">)</span> list available snapshots:
<span class="o">=></span> <span class="c"># zfs list -t snap -r pool/dataset</span>
<span class="o">==</span>
<span class="o">==</span> You can <span class="nb">set </span>properties on pools/datasets to <span class="nb">enable</span>/disable
<span class="o">==</span> specific snapshots <span class="o">(</span><span class="k">if </span><span class="nb">unset</span>, <span class="k">then </span>it evaluates to <span class="s2">"true"</span><span class="o">)</span>:
<span class="o">=></span> <span class="c"># zfs set com.sun:auto-snapshot=true pool/dataset</span>
<span class="o">=></span> <span class="c"># zfs set com.sun:auto-snapshot:hourly=false pool/dataset</span></code></pre></figure>
<p>This automated system used to use cronjobs to run the included
<code class="highlighter-rouge">zfs-auto-snapshot</code> script, but it looks like back in September 2014 the
cronjobs were swapped out for <a href="https://wiki.archlinux.org/index.php/Systemd/Timers">systemd
timers</a>.</p>
<p>If you are unfamiliar with systemd timers (like I was when I came across this)
they are simply timers which control a systemd service. You can view active
timers with <code class="highlighter-rouge">systemctl list-timers</code>. You will notice that when we installed
<code class="highlighter-rouge">zfs-auto-snapshot-git</code> a set of <code class="highlighter-rouge">.service</code> and <code class="highlighter-rouge">.timer</code> files were installed
in <code class="highlighter-rouge">/usr/lib/systemd/system</code>:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">zfs-auto-snapshot-daily.service
zfs-auto-snapshot-daily.timer
zfs-auto-snapshot-frequent.service
zfs-auto-snapshot-frequent.timer
zfs-auto-snapshot-hourly.service
zfs-auto-snapshot-hourly.timer
zfs-auto-snapshot-monthly.service
zfs-auto-snapshot-monthly.timer
zfs-auto-snapshot-weekly.service
zfs-auto-snapshot-weekly.timer</code></pre></figure>
<p>Each <code class="highlighter-rouge">.service</code> and <code class="highlighter-rouge">.timer</code> pair is for a time interval for making snapshots.
Let’s take a look at the hourly pair.</p>
<h3 id="zfs-auto-snapshot-hourlytimer">zfs-auto-snapshot-hourly.timer</h3>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># See systemd.timers and systemd.time manpages for details</span>
<span class="o">[</span>Unit]
<span class="nv">Description</span><span class="o">=</span>ZFS hourly snapshot timer
<span class="o">[</span>Timer]
<span class="nv">OnCalendar</span><span class="o">=</span>hourly
<span class="nv">Persistent</span><span class="o">=</span><span class="nb">true</span>
<span class="o">[</span>Install]
<span class="nv">WantedBy</span><span class="o">=</span>timers.target</code></pre></figure>
<h3 id="zfs-auto-snapshot-hourlyservice">zfs-auto-snapshot-hourly.service</h3>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">[</span>Unit]
<span class="nv">Description</span><span class="o">=</span>ZFS hourly snapshot service
<span class="o">[</span>Service]
<span class="nv">ExecStart</span><span class="o">=</span>/usr/bin/zfs-auto-snapshot <span class="nt">--prefix</span><span class="o">=</span>znap <span class="nt">--quiet</span> <span class="nt">--syslog</span> <span class="nt">--label</span><span class="o">=</span>hourly <span class="nt">--keep</span><span class="o">=</span>24 //</code></pre></figure>
<p>As you can see the files are fairly simple. <code class="highlighter-rouge">Persistent=true</code> in the <code class="highlighter-rouge">.timer</code>
file will run the job if we happened to miss the last time it should have run,
say if your computer was off or suspended, like an anacron job. The <code class="highlighter-rouge">.service</code>
file just runs the the <code class="highlighter-rouge">zfs-auto-snapshot</code> script, labels it as <code class="highlighter-rouge">hourly</code> and
keeps 24 snapshots. The <code class="highlighter-rouge">//</code> refers to all storage nodes.</p>
<p>To enable automated snapshots we will first have to set an auto-snapshot
parameter as stated in the installation output:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">zfs <span class="nb">set </span>com.sun:auto-snapshot<span class="o">=</span><span class="nb">true </span>pool/dataset</code></pre></figure>
<p>Then we just have to enable whatever timer(s) we want, this will enable an
automated backup at that interval:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">systemctl <span class="nb">enable </span>zfs-auto-snapshot-daily.timer</code></pre></figure>
<p>You can always manually run <code class="highlighter-rouge">zfs-auto-snapshot</code> which will produce a single
backup. I have setup an hourly, daily, weekly and monthly auto-snapshot which
has been running now for two months and everything seems to be going smoothly.</p>
<p>If you’d like to read more about snapshots I’d suggest checking out this page:
<a href="https://pthree.org/2012/12/19/zfs-administration-part-xii-snapshots-and-clones/">https://pthree.org/2012/12/19/zfs-administration-part-xii-snapshots-and-clones/</a>.</p>Brian Koopmanbrian.koopman@yale.eduShort guide on setting up zfs-auto-snapshot-git. Also learn a little about systemd timers.ZFS: Scrubbing with an Asyncronous Cronjob2015-05-17T18:52:45-04:002015-05-17T18:52:45-04:00https://briankoopman.com/zfs-scrubbing-with-an-asyncronous-cronjob<p>If you’re following along, we’ve covered <a href="http://blog.briankoopman.com/moving/">setting up a ZFS
mirror</a> and <a href="http://blog.briankoopman.com/upgrading-zfs/">upgrading
ZFS</a>. Today we’ll discuss
scrubbing our zpool.</p>
<h2 id="scrubbing">Scrubbing</h2>
<p>ZFS scrubbing is a process which detects and corrects silent data errors in the
pool. During a scrub a checksum is being calculated and compared to a known
value. If the values differ, then zfs tries to correct the error with good data
in the pool.</p>
<p>We want to avoid silent data corruption, so we should scrub our zpool
regularly. This can be done while the system is running and could easily be
handled with a weekly cronjob. However, I suspend my desktop every night, so if
I used a simple cronjob there would be a chance my computer would be asleep
when the scrub is scheduled to run.</p>
<h2 id="anacron">Anacron</h2>
<p>Enter anacron, which stands for “anac(h)ronistic cron”. Anacron is like cron in
that it schedules regularly occurring events, however, it does not assume the
computer is on during the scheduled time. When the system is turned back on
anacron checks the last time a task was run and compares it to when it should
be run next (sometimes this will be in the past). If it’s been, say, more than
a week since our last zfs scrub, it runs the scrub. On Arch Linux anacron comes
with <a href="https://www.archlinux.org/packages/?name=cronie">cronie</a>.</p>
<h2 id="installation-and-setup">Installation and Setup</h2>
<p>First we need to install <code class="highlighter-rouge">cronie</code>.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">sudo </span>pacman <span class="nt">-S</span> cronie</code></pre></figure>
<p>Now, to setup our zfs scrub we need to add the following line to
<code class="highlighter-rouge">/etc/anacrontab</code>:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># zfs scrub</span>
7 10 scrublotus.weekly zpool scrub lotus</code></pre></figure>
<p>You’ll see the comments in your anacrontab describing the columns, but for
completeness the columns are the period in days, in this case we want to run
the zpool scrub once a week, so I’ve entered 7, the delay for starting the job
in minutes, we have set this to 10, the job-identifier, which is a unique name
given to the job and is used when storing the last run time for a job, this
could be anything, and finally the command we want to run, <code class="highlighter-rouge">zpool scrub lotus</code>.
You’ll want to enter the name of your zpool where I’ve entered <code class="highlighter-rouge">lotus</code>.</p>
<p>Anacron inserts a random delay to the start of the jobs it runs on top of the
entered base delay in the second column. This is meant to prevent all the jobs
from running at once, potentially putting a sudden large load on the system.</p>
<p>Finally we need to make sure the <code class="highlighter-rouge">cronie.service</code> is running:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">sudo </span>systemctl <span class="nb">enable </span>cronie.service
<span class="nb">sudo </span>systemctl start cronie.service</code></pre></figure>
<p>If <code class="highlighter-rouge">cronie</code> is running you’ll see:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">sudo </span>systemctl status cronie.service
● cronie.service - Periodic Command Scheduler
Loaded: loaded <span class="o">(</span>/usr/lib/systemd/system/cronie.service<span class="p">;</span> enabled<span class="p">;</span> vendor preset: disabled<span class="o">)</span>
Active: active <span class="o">(</span>running<span class="o">)</span> since Sat 2015-05-16 18:37:23 EDT<span class="p">;</span> 19h ago
Main PID: 673 <span class="o">(</span>crond<span class="o">)</span>
CGroup: /system.slice/cronie.service
└─673 /usr/bin/crond <span class="nt">-n</span></code></pre></figure>
<h2 id="scrub-status">Scrub Status</h2>
<p>To view the status of an on going scrub, or when you last finished a scrub you
can run <code class="highlighter-rouge">sudo zpool status</code>, there will be a line with scan results on it
similar to this one.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">scan: scrub repaired 0 <span class="k">in </span>3h2m with 0 errors on Sat May 16 14:15:45 2015</code></pre></figure>
<p>During a scrub there will be a lot of I/O operations occurring, so reading and
writing for normal usage will slow down a bit. I was curious about whether or
not I could suspend or shutdown the computer during a scrub when I first set
this up, as there is a chance I will do so, but I could not find a lot of
information on the topic. I have not had issue with it yet, though I believe I
have suspended during a scrub at least once since I set this up.</p>
<p>Now that we have automated scrubbing setup, one of the last automated
maintenance tasks to setup is automated snapshots of the zpool. I’ll write
about this hopefully next week, though it may get delayed by up to a month, as
I have a rather involved, month long, exam for my degree beginning tomorrow.</p>Brian Koopmanbrian.koopman@yale.eduIf you’re following along, we’ve covered setting up a ZFS mirror and upgrading ZFS. Today we’ll discuss scrubbing our zpool.ZFS: Upgrading2015-05-11T02:37:58-04:002015-05-11T02:37:58-04:00https://briankoopman.com/upgrading-zfs<p>Last week we walked through <a href="http://blog.briankoopman.com/moving/">setting up a ZFS
mirror</a>. This week we’ll talk about
upgrading zfs. Note I’m using Arch Linux, which has a rolling release model, so
I update fairly frequently.</p>
<p>We need to take care when performing updates now because ZFS depends on the
kernel version we are on, so upgrading is not as simple as it used to be. If
the kernel updates, ZFS needs to be updated with a version built on the new
kernel. Since we are only using ZFS on our home directory I think we could get
away with uninstalling ZFS and it’s dependancies, updating, reinstalling
things, then just remount the zpool, but this seems inefficient to me.</p>
<p>I’ve come to learn that there is an <a href="https://wiki.archlinux.org/index.php/Unofficial_user_repositories#archzfs">unofficial user
repository</a>
run by demizer. We will add this to our pacman.conf and use it to perform
upgrades.</p>
<h2 id="adding-an-unofficial-user-repository">Adding an Unofficial User Repository</h2>
<p>We first need to add the unofficial repo to our pacman.conf. Add the following
to <code class="highlighter-rouge">/etc/pacman.conf</code>:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># demizer's ZFS repo</span>
<span class="o">[</span>archzfs]
Server <span class="o">=</span> http://archzfs.com/<span class="nv">$repo</span>/x86_64</code></pre></figure>
<p>Now we need to sign the key.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">sudo </span>pacman-key <span class="nt">-r</span> 0EE7A126
<span class="o">[</span><span class="nb">sudo</span><span class="o">]</span> password <span class="k">for </span>koopman:
gpg: key 0EE7A126: public key <span class="s2">"Jesus Alvarez <jeezusjr@gmail.com>"</span> imported
gpg: Total number processed: 1
gpg: imported: 1
<span class="o">==></span> Updating trust database...
gpg: next trustdb check due at 2016-01-22</code></pre></figure>
<p>We can verify the fingerprint by running the following and checking the last 8
digits of the fingerprint match the ID.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">sudo </span>pacman-key <span class="nt">-f</span> 0EE7A126
<span class="o">[</span><span class="nb">sudo</span><span class="o">]</span> password <span class="k">for </span>koopman:
pub rsa2048/0EE7A126 2012-10-24
Key fingerprint <span class="o">=</span> B18A 9C9F 1E4E EAFF 072D AB9E 5E1A BF24 0EE7 A126
uid <span class="o">[</span> unknown] Jesus Alvarez <jeezusjr@gmail.com>
sub rsa2048/DAB97A2B 2012-10-24</code></pre></figure>
<p>Finally, we’ll sign the key:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">sudo </span>pacman-key <span class="nt">--lsign-key</span> 0EE7A126
-> Locally signing key 0EE7A126...
<span class="o">==></span> Updating trust database...
gpg: 3 marginal<span class="o">(</span>s<span class="o">)</span> needed, 1 <span class="nb">complete</span><span class="o">(</span>s<span class="o">)</span> needed, PGP trust model
gpg: depth: 0 valid: 1 signed: 6 trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: depth: 1 valid: 6 signed: 62 trust: 1-, 0q, 0n, 5m, 0f, 0u
gpg: depth: 2 valid: 62 signed: 5 trust: 62-, 0q, 0n, 0m, 0f, 0u
gpg: next trustdb check due at 2016-01-22</code></pre></figure>
<h2 id="updating">Updating</h2>
<p>Now when we run <code class="highlighter-rouge">pacman -Syy</code> we’ll see the new repo:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">sudo </span>pacman <span class="nt">-Syy</span>
:: Synchronizing package databases...
core 121.4 KiB 826K/s 00:00 <span class="o">[</span><span class="c">################################] 100%</span>
extra 1800.5 KiB 2.55M/s 00:01 <span class="o">[</span><span class="c">################################] 100%</span>
community 2.6 MiB 2.58M/s 00:01 <span class="o">[</span><span class="c">################################] 100%</span>
multilib 125.0 KiB 2.84M/s 00:00 <span class="o">[</span><span class="c">################################] 100%</span>
archzfs 7.6 KiB 136K/s 00:00 <span class="o">[</span><span class="c">################################] 100%</span>
archzfs.sig 287.0 B 0.00B/s 00:00 <span class="o">[</span><span class="c">################################] 100%</span></code></pre></figure>
<p>And since we just ran <code class="highlighter-rouge">pacman -Syy</code> we should update with <code class="highlighter-rouge">pacman -Syu</code>. If you
notice a kernel update you should check that the latest version of ZFS matches.
demizer does a great job of keeping the repo up to date when a new kernel
version is released, but he’s only human, so sometimes the repo might not be
caught up yet. In this case I usually just wait a little bit.</p>
<p>Now upgrading is nearly as easy as it used to be. I hope your ZFS setup is
working as smoothly as mine has been. Next week I’ll cover asynchronous
automated scrubbing of a zpool.</p>Brian Koopmanbrian.koopman@yale.eduNow that we have installed ZFS we need to think about how to keep it up to date. We add the unofficial archzfs repo to do so.ZFS: Setting up a ZFS Mirror2015-05-02T22:54:48-04:002015-05-02T22:54:48-04:00https://briankoopman.com/moving<p>Lately I have been performing upgrades to the hard drives in my computers. The
hard drive related project today is setting up a ZFS mirror on my desktop which
will contain my <code class="highlighter-rouge">/home</code> directory.</p>
<h1 id="zfs-intro">ZFS Intro</h1>
<p>I assume if you’re reading this then might you know was ZFS is, but in case you
don’t, ZFS is an advanced file system developed by Sun Microsystems. It
provides a lot of really attractive features for providing redundancy in your
data. ZFS allows you to integrate many disks into a “zpool” for storage. There
are many different configurations one could setup, but I’ll just focus on a
simple ZFS mirror, where I have two disks, each of which will contain the same
data, hence the name ‘mirror’.</p>
<p>This post is going to cover my experience setting up a ZFS mirror, if you want
to know more about ZFS (and trust me, there is a lot more to know), I’ve really
found <a href="https://pthree.org/2012/04/17/install-zfs-on-debian-gnulinux/">this</a>
series from Aaron Toponce to be the best guide. That and the ArchWiki <a href="https://wiki.archlinux.org/index.php/ZFS">ZFS
post</a> of course.</p>
<h2 id="motivation">Motivation</h2>
<p>So why setup a ZFS mirror? Well, I’ve always been interested in setting up a
RAID on my desktop. In doing some research on RAID I came across ZFS and it
seemed to me like the better option. I’m pretty paranoid when it comes to
keeping backups of my data, and while something like RAID or ZFS does not
substitute for proper backups, the thought of redundant drives is appealing to
me. Once setup, if we lose a disk due to hardware failure the pool will still
run in a degraded state, giving us time to swap in a new drive.</p>
<h1 id="initial-hdd-configuration">Initial HDD Configuration</h1>
<p>If you’re reading with the intention of setting up your own ZFS mirror we’re
probably starting in different initial configurations. My setup at the start
involves two SSDs and two HDDs. The two SSDs are 120 GB each; one boots
Windows, the other contains <code class="highlighter-rouge">/</code>, <code class="highlighter-rouge">/boot/</code> and <code class="highlighter-rouge">swap</code> for my Arch linux
installation. (It probably shouldn’t have swap on it, but I’ll address that in
another post.)</p>
<p>The two HDDs are different capacities. Their is a 640GB drive, which holds my
<code class="highlighter-rouge">/home</code> directory, and a 2TB drive which is split into two 1TB partitions, one
for Windows, one for linux, both for storage.</p>
<p>I did a lot of data shuffling to prep for setting up this ZFS mirror with the
2TB drive and a new 2TB drive I just purchased. So, I’m starting from a
configuration where I’m booting from the linux SSD and I have no home directory
(It’s backed up externally.) This gives us two empty 2TB drives to create the
mirror on.</p>
<p>To get started we’ll need to install ZFS, and on Arch this is going to be from
the AUR. Install how you want, but I use <code class="highlighter-rouge">yaourt</code>:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">yaourt <span class="nt">-S</span> zfs-git</code></pre></figure>
<p>The kernel modules for ZFS are tied to a specific kernel, which required a
downgrade for me at the time I installed. Once you exit dependency hell zfs
will be installed. Before we jump into making a zpool we should take a second
to think about the drives we’re using.</p>
<h2 id="advanced-format-drives">Advanced Format Drives</h2>
<p>Something I learned about while setting this up (learning new things is another
large part of the motivation for me here) is Advanced Format drives. Older
drives will store data in 512 byte sectors. New “Advanced Format” drives store
data in 4096 byte sectors.</p>
<p>Where this comes in with ZFS is a potential performance loss if you don’t
explicitly define the sector size when making the zpool. This is pointed out in
the <a href="https://wiki.archlinux.org/index.php/ZFS#Advanced_format_disks">ArchWiki</a>,
and can be done by providing the <code class="highlighter-rouge">-o ashift=12</code> option.</p>
<h3 id="mixing-drives-types">Mixing Drives Types</h3>
<p>Part of what made me read more and more about Advanced Format drives is the
fact that I didn’t know how to tell if the drives I had were Advanced Format or
not and if the two could be mixed in a zpool. They’re both WD Black drives but
they were bought almost two years apart. Fortunately, I found
<a href="http://www.wdc.com/wdproducts/library/other/2579-001028.pdf">this</a> (pdf) model
number format guide which details the letters in each model number.</p>
<p>I have a new WD2003FZEX drive and an old WD2002FAEX drive. The difference being
the new drive is Advanced Format and the old drive isn’t. So, can we mix the
two?</p>
<p>Several people on a few forums suggest forcing the 4K sectors when mixing
drives, as not much performance is lost in forcing 512 byte sectors to align to
4096 byte sector, but there is definite performance lost the other way. So I’ll
be throwing that <code class="highlighter-rouge">-o ashift=12</code> option when I create my zpool. If you’re using
Advanced Format drives you should do the same.</p>
<h1 id="setting-up-the-zpool">Setting Up the zpool</h1>
<p>I wiped the data off my old drive and so both drives are unformatted (i.e. no
file system on them). To see all your drives we can use parted,</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">[</span>root@example ~]# parted <span class="nt">--list</span>
Model: ATA WDC WD2003FZEX-0 <span class="o">(</span>scsi<span class="o">)</span>
Disk /dev/sda: 2000GB
Sector size <span class="o">(</span>logical/physical<span class="o">)</span>: 512B/4096B
Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags
1 1049kB 2000GB 2000GB zfs
9 2000GB 2000GB 8389kB
Model: ATA WDC WD6401AALS-0 <span class="o">(</span>scsi<span class="o">)</span>
Disk /dev/sdb: 640GB
Sector size <span class="o">(</span>logical/physical<span class="o">)</span>: 512B/512B
Partition Table: msdos
Disk Flags:
Number Start End Size Type File system Flags
1 1049kB 640GB 640GB primary ntfs
Model: ATA WDC WD2002FAEX-0 <span class="o">(</span>scsi<span class="o">)</span>
Disk /dev/sdc: 2000GB
Sector size <span class="o">(</span>logical/physical<span class="o">)</span>: 512B/512B
Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags
1 1049kB 2000GB 2000GB zfs
9 2000GB 2000GB 8389kB
Model: ATA OCZ-VECTOR <span class="o">(</span>scsi<span class="o">)</span>
Disk /dev/sdd: 128GB
Sector size <span class="o">(</span>logical/physical<span class="o">)</span>: 512B/512B
Partition Table: msdos
Disk Flags:
Number Start End Size Type File system Flags
1 1049kB 128GB 128GB primary ntfs boot
Model: ATA OCZ-VECTOR <span class="o">(</span>scsi<span class="o">)</span>
Disk /dev/sde: 128GB
Sector size <span class="o">(</span>logical/physical<span class="o">)</span>: 512B/512B
Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags
1 1049kB 3146kB 2097kB bios_grub
2 3146kB 527MB 524MB ext4 boot, esp
3 527MB 17.7GB 17.2GB linux-swap<span class="o">(</span>v1<span class="o">)</span>
4 17.7GB 128GB 110GB ext4 msftdata</code></pre></figure>
<p>I’m looking for my only two 2TB drives, which for me were <code class="highlighter-rouge">/dev/sda</code> and
<code class="highlighter-rouge">/dev/sdc</code>, your drives will probably be different. We’ll want the disk id’s to
make the zpool, which we can get just by looking in <code class="highlighter-rouge">/dev/</code>,</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">[</span>root@example ~]# <span class="nb">ls</span> <span class="nt">-lah</span> /dev/disk/by-id/
total 0
drwxr-xr-x 2 root root 660 Mar 24 22:43 <span class="nb">.</span>
drwxr-xr-x 7 root root 140 Mar 24 22:39 ..
lrwxrwxrwx 1 root root 9 Mar 24 22:04 ata-ASUS_DRW-24B1ST_a_B5D0CL354810 -> ../../sr0
lrwxrwxrwx 1 root root 9 Mar 24 22:43 ata-OCZ-VECTOR_OCZ-63C1889KG99WW96L -> ../../sdd
lrwxrwxrwx 1 root root 10 Mar 24 22:43 ata-OCZ-VECTOR_OCZ-63C1889KG99WW96L-part1 -> ../../sdd1
lrwxrwxrwx 1 root root 9 Mar 24 22:43 ata-OCZ-VECTOR_OCZ-8MGKY5RPKL9I42JV -> ../../sde
lrwxrwxrwx 1 root root 10 Mar 24 22:43 ata-OCZ-VECTOR_OCZ-8MGKY5RPKL9I42JV-part1 -> ../../sde1
lrwxrwxrwx 1 root root 10 Mar 24 22:43 ata-OCZ-VECTOR_OCZ-8MGKY5RPKL9I42JV-part2 -> ../../sde2
lrwxrwxrwx 1 root root 10 Mar 24 22:43 ata-OCZ-VECTOR_OCZ-8MGKY5RPKL9I42JV-part3 -> ../../sde3
lrwxrwxrwx 1 root root 10 Mar 24 22:43 ata-OCZ-VECTOR_OCZ-8MGKY5RPKL9I42JV-part4 -> ../../sde4
lrwxrwxrwx 1 root root 9 Mar 24 22:43 ata-WDC_WD2002FAEX-007BA0_WD-WCAY00770606 -> ../../sdc
lrwxrwxrwx 1 root root 10 Mar 24 22:43 ata-WDC_WD2002FAEX-007BA0_WD-WCAY00770606-part1 -> ../../sdc1
lrwxrwxrwx 1 root root 10 Mar 24 22:43 ata-WDC_WD2002FAEX-007BA0_WD-WCAY00770606-part9 -> ../../sdc9
lrwxrwxrwx 1 root root 9 Mar 24 22:43 ata-WDC_WD2003FZEX-00Z4SA0_WD-WMC5H0DAU37A -> ../../sda
lrwxrwxrwx 1 root root 10 Mar 24 22:43 ata-WDC_WD2003FZEX-00Z4SA0_WD-WMC5H0DAU37A-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Mar 24 22:43 ata-WDC_WD2003FZEX-00Z4SA0_WD-WMC5H0DAU37A-part9 -> ../../sda9
lrwxrwxrwx 1 root root 9 Mar 24 22:43 ata-WDC_WD6401AALS-00J7B0_WD-WMATV7730189 -> ../../sdb
lrwxrwxrwx 1 root root 10 Mar 24 22:43 ata-WDC_WD6401AALS-00J7B0_WD-WMATV7730189-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 9 Mar 24 22:43 wwn-0x50014ee0579c8eef -> ../../sdb
lrwxrwxrwx 1 root root 10 Mar 24 22:43 wwn-0x50014ee0579c8eef-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 9 Mar 24 22:43 wwn-0x50014ee0ae9998c9 -> ../../sda
lrwxrwxrwx 1 root root 10 Mar 24 22:43 wwn-0x50014ee0ae9998c9-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Mar 24 22:43 wwn-0x50014ee0ae9998c9-part9 -> ../../sda9
lrwxrwxrwx 1 root root 9 Mar 24 22:43 wwn-0x50014ee25cdf92e3 -> ../../sdc
lrwxrwxrwx 1 root root 10 Mar 24 22:43 wwn-0x50014ee25cdf92e3-part1 -> ../../sdc1
lrwxrwxrwx 1 root root 10 Mar 24 22:43 wwn-0x50014ee25cdf92e3-part9 -> ../../sdc9
lrwxrwxrwx 1 root root 9 Mar 24 22:43 wwn-0x5e83a972745093b5 -> ../../sde
lrwxrwxrwx 1 root root 10 Mar 24 22:43 wwn-0x5e83a972745093b5-part1 -> ../../sde1
lrwxrwxrwx 1 root root 10 Mar 24 22:43 wwn-0x5e83a972745093b5-part2 -> ../../sde2
lrwxrwxrwx 1 root root 10 Mar 24 22:43 wwn-0x5e83a972745093b5-part3 -> ../../sde3
lrwxrwxrwx 1 root root 10 Mar 24 22:43 wwn-0x5e83a972745093b5-part4 -> ../../sde4
lrwxrwxrwx 1 root root 9 Mar 24 22:43 wwn-0x5e83a97514d28d2a -> ../../sdd
lrwxrwxrwx 1 root root 10 Mar 24 22:43 wwn-0x5e83a97514d28d2a-part1 -> ../../sdd1</code></pre></figure>
<p>So what we want to note is that <code class="highlighter-rouge">sda</code> has the id
<code class="highlighter-rouge">ata-WDC_WD2003FZEX-00Z4SA0_WD-WMC5H0DAU37A</code> and <code class="highlighter-rouge">sdc</code> has the id
<code class="highlighter-rouge">ata-WDC_WD2002FAEX-007BA0_WD-WCAY00770606</code>.</p>
<p>We’ll need this info to make the zpool, which we can do with the <code class="highlighter-rouge">zpool create</code>
command:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">[</span>koopman@honey etc]<span class="nv">$ </span><span class="nb">sudo </span>zpool create <span class="nt">-f</span> <span class="nt">-o</span> <span class="nv">ashift</span><span class="o">=</span>12 <span class="nt">-m</span> /home lotus mirror ata-WDC_WD2003FZEX-00Z4SA0_WD-WMC5H0DAU37A ata-WDC_WD2002FAEX-007BA0_WD-WCAY00770606</code></pre></figure>
<p>The options I’ve used are <code class="highlighter-rouge">-f</code>, which forces the use of vdevs (<a href="https://pthree.org/2012/12/04/zfs-administration-part-i-vdevs/">virtual
devices</a>), ours
will be a mirror, <code class="highlighter-rouge">-o ashift=12</code>, which forces the 4096 byte sectors for
advanced format drives, and <code class="highlighter-rouge">-m /home</code>, which sets the mountpoint for the
zpool. The rest of the command contains <code class="highlighter-rouge">lotus</code>, the name of my new pool,
<code class="highlighter-rouge">mirror</code>, the type of ZFS pool we wanted to create, and the two disk id’s.</p>
<h2 id="making-a-dataset">Making a dataset</h2>
<p>Now that we have the zpool we will create a dataset for our home directory:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">sudo </span>zfs create lotus/koopman</code></pre></figure>
<p>Datasets allow you to do such things as setting disk usage quotas and
maintaining individual snapshots per dataset.</p>
<p>Note that you’ll probably need to chown the new directory to use it,</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">sudo chown </span>koopman koopman/</code></pre></figure>
<p>Now we can start using our new zpool!</p>
<h1 id="finishing-up">Finishing Up</h1>
<p>This was a lot easier than I pictured years ago when thinking I’d like to setup
a RAID, granted we used ZFS, which seems to be a lot nicer in many respects.</p>
<p>To wrap up we’ll do some quick tweaks. The first of which will be auto-mounting
at boot. The zfs daemon is capable of loading a zpool and mounting it at boot.
In order to do this we need to setup a cache file, then enable the zfs daemon.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">zpool <span class="nb">set </span><span class="nv">cachefile</span><span class="o">=</span>/etc/zfs/zpool.cache lotus
systemctl <span class="nb">enable </span>zfs.target</code></pre></figure>
<p>Next, we’re going to enable relatime and turn on lz4 compression to decrease
the number of superfluous writes we do to the zpool and to compress our data.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">zfs <span class="nb">set </span><span class="nv">relatime</span><span class="o">=</span>on lotus
zfs <span class="nb">set </span><span class="nv">compression</span><span class="o">=</span>lz4 lotus</code></pre></figure>
<p>And that’s it, we’ve setup a ZFS mirror as our home directory! You can check
out your new zpool with a simple <code class="highlighter-rouge">zpool status</code>:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">sudo </span>zpool status
<span class="o">[</span><span class="nb">sudo</span><span class="o">]</span> password <span class="k">for </span>koopman:
pool: lotus
state: ONLINE
scan: scrub repaired 0 <span class="k">in </span>2h55m with 0 errors on Wed Apr 15 23:22:56 2015
config:
NAME STATE READ WRITE CKSUM
lotus ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-WDC_WD2003FZEX-00Z4SA0_WD-WMC5H0DAU37A ONLINE 0 0 0
ata-WDC_WD2002FAEX-007BA0_WD-WCAY00770606 ONLINE 0 0 0
errors: No known data errors</code></pre></figure>
<p>(Note: On a fresh new pool you won’t see the “scan” line, this comes in from
scrubbing your zpool, something that I’ll describe how to automate in a later
post.)</p>
<p>Then all we need to do is <code class="highlighter-rouge">rsync</code> our data to our home directory from our
backups and enjoy the new disk redundancy.</p>
<h2 id="further-adjustments">Further Adjustments</h2>
<p>I really setup this pool about a month ago. There are many more things you can
and should do with your zpool, such as having automated snapshots and scrubbing
your zpool regularly. Setting all these things up at once takes a while on the
first time through, and I was planning on including it all in this post. As a
result, this already lengthy post was becoming way too long. So expect to see
more ZFS related posts in the near future.</p>Brian Koopmanbrian.koopman@yale.eduMy first time setting up a zpool. We walk through setting up a basic ZFS mirror and discuss Advanced Format drives.Upgrading to Gigabit and Testing it with netcat2015-03-29T02:43:42-04:002015-03-29T02:43:42-04:00https://briankoopman.com/computer-upgrades<p>When I setup the network connection to the room in my apartment I bought a <code class="highlighter-rouge">100
Mbps</code> switch, choosing not to spend the extra $20 for gigabit. Today I’m
upgrading to a gigabit switch, having realized that the <code class="highlighter-rouge">100 Mbps</code> switch is
what’s limiting local traffic between my desktop and server.</p>
<p>The other part of my motivation comes from setting up a new ZFS mirror on my
desktop and wanting to transfer data off my sever to the mirror over the
network. I then plan on streaming media from the desktop to the server, which
is being repurposed to a HTPC (with a much smaller SSD instead of the failing
2TB drive in it now).</p>
<p>Before I swapped in the new switch I wanted to test the network transfer
speeds, something I have never really done before, just so I can see the speed
increase associated with the new switch.</p>
<p>I found <a href="http://deice.daug.net/netcat_speed.html">this</a> guide helpful, and
we’re basically following along here.</p>
<h2 id="netcat">Netcat</h2>
<p>To test the <code class="highlighter-rouge">100 Mbps</code> connection we’ll use <code class="highlighter-rouge">netcat</code>. I am testing from my
desktop to my server. We’ll need netcat on both computers we want to test. On
the server (which runs Ubuntu 12.04) we’ll need to install the
<code class="highlighter-rouge">netcat-traditional</code> package:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">sudo </span>apt-get <span class="nb">install </span>netcat-traditional</code></pre></figure>
<p>We’ll also need netcat on the desktop (running Arch linux). The package is
called <code class="highlighter-rouge">gnu-netcat</code>:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">sudo </span>pacman <span class="nt">-S</span> gnu-netcat</code></pre></figure>
<h2 id="100-mbps-test">100 Mbps Test</h2>
<p>To start we’ll need to setup the listening part of the connection on the
server:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>nc.traditional <span class="nt">-v</span> <span class="nt">-v</span> <span class="nt">-l</span> <span class="nt">-n</span> <span class="nt">-p</span> 2222 <span class="o">></span> /dev/null
listening on <span class="o">[</span>any] 2222 ...</code></pre></figure>
<p>The flags here get us a very verbose output (the two <code class="highlighter-rouge">-v</code> options), <code class="highlighter-rouge">-l</code> sets
up listen mode, <code class="highlighter-rouge">-n</code> tells us we want to user numeric-only IP addresses, not
DNS, and <code class="highlighter-rouge">-p 2222</code> tells netcat to listen on port 2222. <code class="highlighter-rouge">> /dev/null</code> redirects
all the traffic we send to <code class="highlighter-rouge">/dev/null</code>.</p>
<p>Then, on the desktop, we can send some traffic to the server:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">time yes</span>|nc <span class="nt">-v</span> <span class="nt">-v</span> <span class="nt">-n</span> 192.168.1.7 2222 <span class="o">></span> /dev/null
192.168.1.7 2222 <span class="o">(</span>EtherNet-IP-1<span class="o">)</span> open
^CExiting.
Total received bytes: 0
Total sent bytes: 111M <span class="o">(</span>110594432<span class="o">)</span>
real 0m9.398s
user 0m1.960s
sys 0m0.623s</code></pre></figure>
<p>This uses <code class="highlighter-rouge">time</code> to track how long we send traffic for. You’ll want to stop
this after about 10 seconds by hitting <code class="highlighter-rouge">ctrl+c</code>. (If you don’t it’s no big
deal, you might just actually wrap the integer count on received traffic,
making it more difficult to calculate speeds.) On the server we’ll see a
message like this:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">connect to <span class="o">[</span>192.168.1.7] from <span class="o">(</span>UNKNOWN<span class="o">)</span> <span class="o">[</span>192.168.1.112] 43341
sent 0, rcvd 110498864 : Connection reset by peer</code></pre></figure>
<p>Now we just need to calculate our resulting speeds. You’ll notice in the sent
traffic the units are bytes. So we’ll take our <code class="highlighter-rouge">110498864 bytes</code> and multiply
by <code class="highlighter-rouge">8 bits/byte</code> and divide by the <code class="highlighter-rouge">9.398s</code> to get <code class="highlighter-rouge">94061599.5 bytes/s</code>, or <code class="highlighter-rouge">94
Mbps</code>.</p>
<p>This is in line with the <code class="highlighter-rouge">100 Mbps</code> switch that I have right now. Let’s swap in
the Gpbs switch.</p>
<h2 id="1000-mbps-test">1000 Mbps Test</h2>
<p>The switch I got was this
<a href="http://www.newegg.com/Product/Product.aspx?Item=N82E16833122128">Netgear</a> 5
Port Gigabit switch from Newegg. They were running a deal for $19.99 with mail
in rebate, so I jumped on it.</p>
<p>The switch indicates that both lights surrounding the ports will illuminate
with gigabit traffic, and upon plugging everything in, they are both blinking,
so I expect to see gigabit speeds.</p>
<p>Running a test we get the following results:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>nc.traditional <span class="nt">-v</span> <span class="nt">-v</span> <span class="nt">-l</span> <span class="nt">-n</span> <span class="nt">-p</span> 2222 <span class="o">></span> /dev/null
listening on <span class="o">[</span>any] 2222 ...
connect to <span class="o">[</span>192.168.1.7] from <span class="o">(</span>UNKNOWN<span class="o">)</span> <span class="o">[</span>192.168.1.112] 47939
sent 0, rcvd 840984576</code></pre></figure>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">time yes</span>|nc <span class="nt">-v</span> <span class="nt">-v</span> <span class="nt">-n</span> 192.168.1.7 2222 <span class="o">></span>/dev/null
192.168.1.7 2222 <span class="o">(</span>EtherNet-IP-1<span class="o">)</span> open
^CExiting.
Total received bytes: 0
Total sent bytes: 870M <span class="o">(</span>869988352<span class="o">)</span>
real 0m10.207s
user 0m9.040s
sys 0m5.110s</code></pre></figure>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="o">>>></span> <span class="mi">840984576</span><span class="o">*</span><span class="mi">8</span><span class="o">/</span><span class="mf">9.780</span><span class="o">/</span><span class="mf">1000000.</span>
<span class="mf">687.9219435582822</span></code></pre></figure>
<p>A solid <code class="highlighter-rouge">687.9 Mbps</code>, much better than our previous <code class="highlighter-rouge">94 Mbps</code>. Not exactly
<code class="highlighter-rouge">1000 Mbps</code> that the switch can do, but I’m much happier with it. Perhaps
something else is limiting me now, though I’m not sure what it is, as the
router is also gigabit wired.</p>
<h2 id="repeatability">Repeatability</h2>
<p>Just to check repeatability I tried the 100 Mbps switch connection a day after
this first test, below are the results:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>nc.traditional <span class="nt">-v</span> <span class="nt">-v</span> <span class="nt">-l</span> <span class="nt">-n</span> <span class="nt">-p</span> 2222 <span class="o">></span> /dev/null
listening on <span class="o">[</span>any] 2222 ...
connect to <span class="o">[</span>192.168.1.7] from <span class="o">(</span>UNKNOWN<span class="o">)</span> <span class="o">[</span>192.168.1.112] 47495
sent 0, rcvd 251576544 : Connection reset by peer</code></pre></figure>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">time yes</span>|nc <span class="nt">-v</span> <span class="nt">-v</span> <span class="nt">-n</span> 192.168.1.7 2222 <span class="o">></span>/dev/null
192.168.1.7 2222 <span class="o">(</span>EtherNet-IP-1<span class="o">)</span> open
^CExiting.
Total received bytes: 0
Total sent bytes: 252M <span class="o">(</span>251669216<span class="o">)</span>
real 0m21.381s
user 0m5.053s
sys 0m1.310s</code></pre></figure>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="o">>>></span> <span class="mi">251576544</span><span class="o">*</span><span class="mi">8</span><span class="o">/</span><span class="mf">21.381</span><span class="o">/</span><span class="mf">1000000.</span>
<span class="mf">94.13088031429774</span></code></pre></figure>
<p>So a pretty repeatable <code class="highlighter-rouge">94 Mbps</code>.</p>
<p>On the gigabit switch I get some more variability, repeating the test a week or
so after installing it I got as much as <code class="highlighter-rouge">728 Mbps</code>, which makes me suspect
other network traffic on the router is limiting what I can get, though I don’t
know enough about a router handles traffic to all it’s ports to know if that’s
a true statement.</p>
<h1 id="conclusions">Conclusions</h1>
<p>I have put a lot of traffic over the new switch in the past week since I
installed it. I’m definitely appreciating the upgrade in speed. If you know
your switch is limiting your speeds and want a cheap upgrade to boost your
local network speeds I can’t recommend a gigabit switch enough, and Netgear
products have always done well for me.</p>Brian Koopmanbrian.koopman@yale.eduWe prove my new gigabit switch is faster than my old 100 Mbps switch using netcat.Super Mario 64 Emulation on Linux2014-12-30T05:18:20-05:002014-12-30T05:18:20-05:00https://briankoopman.com/super-mario-64-emulation-on-linux<p>A friend of mine has been speedrunning The Legend of Zelda: Ocarina of Time
(OoT) for a while now. Not too long ago a friend of his began speedrunning OoT
as well, but also has been running Super Mario 64 (SM64). OoT didn’t really
appeal to me, but SM64 does. I used to play a lot of NES emulators in high
school and had some N64 emulators, but that was still when those were
relatively new.</p>
<p>I decided that I wanted to try settings up SM64 to run on linux, and stream to
twitch.tv, partly to try speedrunning and partly because getting things
configured to work on linux is always enjoyable for me.</p>
<h1 id="emulator">Emulator</h1>
<p>It began with the choice of emulator, a quick search turned up
<a href="https://code.google.com/p/mupen64plus/">Mupen64Plus</a>, I downloaded the binary
64-bit version, downloaded some dependencies (<code class="highlighter-rouge">minizip 1.2.8-1</code> and
<code class="highlighter-rouge">libpng15</code>), dug up an old rom file for SM64 and it started up no problem!</p>
<h2 id="configuring-the-controller">Configuring the Controller</h2>
<p>A lot of people run on a Wii, and if they do they’re probably using a Gamecube
controller. My friends use a Gamecube controller to USB adapter. Not having a
Gamecube controller here or the adapter I wanted to use my Xbox 360 controller.</p>
<p>It turns out that the 360 controller is working out of the box on Arch Linux,
and Mupen64Plus has a default configuration for it, however, that configuration
is really messed up. The configuration can be a little confusing, as there is a
controller config in <code class="highlighter-rouge">~/.config/mupen64plus</code>, however this is regenerated every
time you start up mupen64plus, so editing this won’t get you much.</p>
<p>When you first download the binary, there is a file, <code class="highlighter-rouge">InputAutoCfg.ini</code>, which
has the presets for the controllers. This gets installed in
<code class="highlighter-rouge">/usr/local/share/mupen64plus</code> after running <code class="highlighter-rouge">./install.sh</code>. We must now edit
that copy to have changes stay across sessions.</p>
<p>In searching around for the above info on where to change configurations for
the 360 controller I found this issue:
https://code.google.com/p/mupen64plus/issues/detail?id=576</p>
<p>Detailed here is someone’s fix for the controller settings. A few of things I
noticed myself right away was that Z and R were mapped to the 360’s L, so in
SM64 you’d always change cameras when trying to long jump, not good.</p>
<h3 id="fixing-the-defaults">Fixing the Defaults</h3>
<p>For completeness here are the default controls. Several fixes need to be
applied, which we’ll walk through step by step.</p>
<figure class="highlight"><pre><code class="language-conf" data-lang="conf">[<span class="n">Microsoft</span> <span class="n">X</span>-<span class="n">Box</span> <span class="m">360</span> <span class="n">pad</span>]
[<span class="n">Win32</span>: <span class="n">Controller</span> (<span class="n">XBOX</span> <span class="m">360</span> <span class="n">For</span> <span class="n">Windows</span>)]
[<span class="n">Win32</span>: <span class="n">XBOX</span> <span class="m">360</span> <span class="n">For</span> <span class="n">Windows</span> (<span class="n">Controller</span>)]
[<span class="n">Win32</span>: <span class="n">XBOX</span> <span class="m">360</span> <span class="n">For</span> <span class="n">Windows</span>]
[<span class="n">Xbox</span> <span class="m">360</span> <span class="n">Wireless</span> <span class="n">Receiver</span>]
[<span class="n">OSX</span>: <span class="n">Wireless</span> <span class="m">360</span> <span class="n">Controller</span>]
[<span class="n">OSX</span>: <span class="n">Controller</span>]
[<span class="n">Linux</span>: <span class="n">Xbox</span> <span class="n">Gamepad</span> (<span class="n">userspace</span> <span class="n">driver</span>)]
<span class="n">plugged</span> = <span class="n">True</span>
<span class="n">plugin</span> = <span class="m">2</span>
<span class="n">mouse</span> = <span class="n">False</span>
<span class="n">AnalogDeadzone</span> = <span class="m">4096</span>,<span class="m">4096</span>
<span class="n">AnalogPeak</span> = <span class="m">32768</span>,<span class="m">32768</span>
<span class="n">DPad</span> <span class="n">R</span> = <span class="n">hat</span>(<span class="m">0</span> <span class="n">Right</span>)
<span class="n">DPad</span> <span class="n">L</span> = <span class="n">hat</span>(<span class="m">0</span> <span class="n">Left</span>)
<span class="n">DPad</span> <span class="n">D</span> = <span class="n">hat</span>(<span class="m">0</span> <span class="n">Down</span>)
<span class="n">DPad</span> <span class="n">U</span> = <span class="n">hat</span>(<span class="m">0</span> <span class="n">Up</span>)
<span class="n">Start</span> = <span class="n">button</span>(<span class="m">7</span>)
<span class="n">Z</span> <span class="n">Trig</span> = <span class="n">axis</span>(<span class="m">2</span>+)
<span class="n">B</span> <span class="n">Button</span> = <span class="n">button</span>(<span class="m">2</span>)
<span class="n">A</span> <span class="n">Button</span> = <span class="n">button</span>(<span class="m">0</span>)
<span class="n">C</span> <span class="n">Button</span> <span class="n">R</span> = <span class="n">axis</span>(<span class="m">4</span>+)
<span class="n">C</span> <span class="n">Button</span> <span class="n">L</span> = <span class="n">axis</span>(<span class="m">4</span>-) <span class="n">button</span>(<span class="m">3</span>)
<span class="n">C</span> <span class="n">Button</span> <span class="n">D</span> = <span class="n">axis</span>(<span class="m">3</span>+) <span class="n">button</span>(<span class="m">1</span>)
<span class="n">C</span> <span class="n">Button</span> <span class="n">U</span> = <span class="n">axis</span>(<span class="m">3</span>-)
<span class="n">R</span> <span class="n">Trig</span> = <span class="n">button</span>(<span class="m">5</span>) <span class="n">axis</span>(<span class="m">2</span>-)
<span class="n">L</span> <span class="n">Trig</span> = <span class="n">button</span>(<span class="m">4</span>)
<span class="n">Mempak</span> <span class="n">switch</span> =
<span class="n">Rumblepak</span> <span class="n">switch</span> =
<span class="n">X</span> <span class="n">Axis</span> = <span class="n">axis</span>(<span class="m">0</span>-,<span class="m">0</span>+)
<span class="n">Y</span> <span class="n">Axis</span> = <span class="n">axis</span>(<span class="m">1</span>-,<span class="m">1</span>+)</code></pre></figure>
<h4 id="problems">Problems</h4>
<ul>
<li>The right trigger is double mapped. <code class="highlighter-rouge">button(5)</code> is the right bumper on the
360 controller, and <code class="highlighter-rouge">axis(2-)</code> is triggered when you release the left trigger
on the 360 controller. We’ll modify this by having just <code class="highlighter-rouge">R Trig = button(5)</code>.</li>
</ul>
<p><strong>Before:</strong></p>
<figure class="highlight"><pre><code class="language-conf" data-lang="conf"><span class="n">R</span> <span class="n">Trig</span> = <span class="n">button</span>(<span class="m">5</span>) <span class="n">axis</span>(<span class="m">2</span>-)</code></pre></figure>
<p><strong>After:</strong></p>
<figure class="highlight"><pre><code class="language-conf" data-lang="conf"><span class="n">R</span> <span class="n">Trig</span> = <span class="n">button</span>(<span class="m">5</span>)</code></pre></figure>
<ul>
<li>The C-Stick is backwards, in the sense that vertical movement corresponds to
horizontal directions. So we want to swap the 3’s and 4’s here. Button’s 3
and 1 are Y and B, respectively. I’m still not sure if I’ll use the
<code class="highlighter-rouge">button(3)</code> and <code class="highlighter-rouge">button(1)</code> mappings, so I’ll leave them as is for now.</li>
</ul>
<p><strong>Before:</strong></p>
<figure class="highlight"><pre><code class="language-conf" data-lang="conf"><span class="n">C</span> <span class="n">Button</span> <span class="n">R</span> = <span class="n">axis</span>(<span class="m">4</span>+)
<span class="n">C</span> <span class="n">Button</span> <span class="n">L</span> = <span class="n">axis</span>(<span class="m">4</span>-) <span class="n">button</span>(<span class="m">3</span>)
<span class="n">C</span> <span class="n">Button</span> <span class="n">D</span> = <span class="n">axis</span>(<span class="m">3</span>+) <span class="n">button</span>(<span class="m">1</span>)
<span class="n">C</span> <span class="n">Button</span> <span class="n">U</span> = <span class="n">axis</span>(<span class="m">3</span>-)</code></pre></figure>
<p><strong>After:</strong></p>
<figure class="highlight"><pre><code class="language-conf" data-lang="conf"><span class="n">C</span> <span class="n">Button</span> <span class="n">R</span> = <span class="n">axis</span>(<span class="m">3</span>+)
<span class="n">C</span> <span class="n">Button</span> <span class="n">L</span> = <span class="n">axis</span>(<span class="m">3</span>-) <span class="n">button</span>(<span class="m">3</span>)
<span class="n">C</span> <span class="n">Button</span> <span class="n">D</span> = <span class="n">axis</span>(<span class="m">4</span>+) <span class="n">button</span>(<span class="m">1</span>)
<span class="n">C</span> <span class="n">Button</span> <span class="n">U</span> = <span class="n">axis</span>(<span class="m">4</span>-)</code></pre></figure>
<ul>
<li>Finally, the D-Pad, I’m not sure what <code class="highlighter-rouge">hat(0 Right)</code> is, but it’s not doing
anything for the D-Pad. We’ll add that functionality.</li>
</ul>
<p><strong>Before:</strong></p>
<figure class="highlight"><pre><code class="language-conf" data-lang="conf"><span class="n">DPad</span> <span class="n">R</span> = <span class="n">hat</span>(<span class="m">0</span> <span class="n">Right</span>)
<span class="n">DPad</span> <span class="n">L</span> = <span class="n">hat</span>(<span class="m">0</span> <span class="n">Left</span>)
<span class="n">DPad</span> <span class="n">D</span> = <span class="n">hat</span>(<span class="m">0</span> <span class="n">Down</span>)
<span class="n">DPad</span> <span class="n">U</span> = <span class="n">hat</span>(<span class="m">0</span> <span class="n">Up</span>)</code></pre></figure>
<p><strong>After:</strong></p>
<figure class="highlight"><pre><code class="language-conf" data-lang="conf"><span class="n">DPad</span> <span class="n">R</span> = <span class="n">hat</span>(<span class="m">0</span> <span class="n">Right</span>) <span class="n">button</span>(<span class="m">12</span>)
<span class="n">DPad</span> <span class="n">L</span> = <span class="n">hat</span>(<span class="m">0</span> <span class="n">Left</span>) <span class="n">button</span>(<span class="m">11</span>)
<span class="n">DPad</span> <span class="n">D</span> = <span class="n">hat</span>(<span class="m">0</span> <span class="n">Down</span>) <span class="n">button</span>(<span class="m">14</span>)
<span class="n">DPad</span> <span class="n">U</span> = <span class="n">hat</span>(<span class="m">0</span> <span class="n">Up</span>) <span class="n">button</span>(<span class="m">13</span>)</code></pre></figure>
<p>Now, I’m not actually sure the D-Pad does anything in SM64, it was pretty
useless on the N64 controller originally, so I’m not too worried about it. Even
with these configurations, it’s not doing anything.</p>
<h1 id="casting-software">Casting Software</h1>
<p>So all I basically looked at here was the Arch Wiki page for
<a href="https://wiki.archlinux.org/index.php/Streaming_using_twitch.tv">streaming</a>,
which resulted in me downloading
<a href="http://sourceforge.net/projects/castawesome/">castawesome</a> through the
<a href="https://aur.archlinux.org/packages/castawesome/">AUR</a>. The AUR dependencies
had things taken care of and it ran no problem. Someone on a twitch chat had
suggested using ffmpeg directly, which is also detailed on the Arch Wiki.</p>
<h2 id="timers">Timers</h2>
<p>The last step was a timer. I tried <a href="http://jenmaarai.com/llanfair/">Llanfair</a>,
as it’s written in Java and had a jar file I could run natively. This had some
issues though, the first of which I noticed was when moving it around it would
glitch out and move where ever it wanted to. Then I started having issues with
text clipping, so I jumped ship and moved to trying to get
<a href="http://livesplit.org/downloads/">LiveSplit</a> working with WINE.</p>
<h3 id="livesplit-and-wine">LiveSplit and WINE</h3>
<p>This was probably the most involved step, as I had to install WINE, then figure
out dependencies for LiveSplit. I’ll assume you have WINE installed on whatever
system you’re running. We’ll then make a 32-bit WINEARCH and a new WINEPREFIX
for livesplit.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">WINEARCH</span><span class="o">=</span>win32 <span class="nv">WINEPREFIX</span><span class="o">=</span>~/.livesplit winecfg</code></pre></figure>
<p>Then we need to install .NET 4.0 and some other things like msxml3 and
corefonts. The <code class="highlighter-rouge">-q</code> option does the install without bothering you.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">WINEARCH</span><span class="o">=</span>win32 <span class="nv">WINEPREFIX</span><span class="o">=</span>~/.livesplit winetricks <span class="nt">-q</span> msxml3 dotnet40 corefonts</code></pre></figure>
<p>Now we can download the latest version of LiveSplit here:
http://livesplit.org/downloads/</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">WINEARCH</span><span class="o">=</span>win32 <span class="nv">WINEPREFIX</span><span class="o">=</span>~/.livesplit wine ~/speedruns/Livesplit_1.4.5/LiveSplit.exe</code></pre></figure>
<p>You should change out the directory I use for the one you’re keeping your .exe
of LiveSplit in.</p>
<h1 id="wrapping-up">Wrapping Up</h1>
<p>That should be it, we have Mupen64Plus emulating a SM64 rom, we fixed some Xbox
360 controller issues that were misconfigured on default, we installed
castawesome to stream to twitch.tv, and finally got LiveSplit working under
WINE. I’ve added some aliases to my <code class="highlighter-rouge">.bashrc</code> file as below, and now just run
three simple commands, hit the “on” button in castawesome, and I’m streaming.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">alias </span><span class="nv">sm64</span><span class="o">=</span><span class="s1">'/home/USER/speedruns/mupen64plus-bundle-linux64-2.0/mupen64plus --resolution 1024x768 /home/USER/speedruns/roms/sm64.z64'</span>
<span class="nb">alias split</span><span class="o">=</span><span class="s1">'WINEARCH=win32 WINEPREFIX=~/.livesplit wine /home/USER/speedruns/Livesplit_1.4.5/LiveSplit.exe'</span></code></pre></figure>
<p>Happy Speed Running!</p>
<h2 id="revisit">Revisit</h2>
<p>So I wrote this as I configured my linux streaming setup, but didn’t publish it
until now. In the time since I have abandoned the linux setup and moved over to
Windows, primarily for the streaming software. castawesome was resulting in
some pretty poor stream quality. The lack of configurability there lead me to
change. And since I play most of my other games on Windows (and often times the
reboot alone stops me from playing games) I’m fine with moving platforms. I
hope someone out there finds this helpful, and if you do, let me know in the
comments.</p>Brian Koopmanbrian.koopman@yale.eduInspired by some friends racing SM64 16-star I setup to play and cast from linux. It works, however poor casting quality drives me to Windows in the end.Updating Ghost2014-12-14T06:33:20-05:002014-12-14T06:33:20-05:00https://briankoopman.com/updating-ghost<p>I will admit, I have not been very good about keeping Ghost up to date. Back in
October I received an email about the big October update, I believe this was
0.5.3. I just updated to the now current version (0.5.6). I have to look up the
process each time I update, so I figure I’ll write about it, that way I’ll have
a solid footing when I update to 0.5.7, perhaps motivating me to keep things up
to date.</p>
<h1 id="finding-support">Finding Support</h1>
<p>I love Ghost as a blogging platform. I can run my own installation of it and
writing posts in it is easy. (Not to mention it looks great.) I, however, find
looking for information on their site a less intuitive experience. There is
something strange to me about the layout, for instance, I know they have a
forum, but there is no link on the main page for it, instead it is under
support.</p>
<p>Anyway, my usual approach is to just turn to Google, but Google often turns up
results from
<a href="http://www.howtoinstallghost.com">http://www.howtoinstallghost.com</a>, which I’m
not sure is actually affiliated with the Ghost team. While I like their goal of
providing easy to use instructions for those looking to just use the platform
without any of the technical details, it lacks some of potentially crucial
info, such as this note about backing up your blog’s data before performing an
upgrade on the actual Ghost <a href="http://support.ghost.org/how-to-upgrade/">support
page</a>:</p>
<blockquote>
<p>Note: You can, if you like, take a copy of your database from content/data
but be warned you should not do this whilst Ghost is running. Please stop it
first.</p>
</blockquote>
<p>I actually ended up doing this while Ghost was still running, thinking it was a
great idea, before I found that page. My data survived, but next time I might
not be so lucky.</p>
<p>That being said, the first thing we should is back up our precious data.</p>
<h1 id="backups">Backups</h1>
<p>As noted <a href="http://support.ghost.org/how-to-upgrade/#backing-up">here</a>, to backup
all we have to do is export a <code class="highlighter-rouge">.json</code> file from the administrative interface of
Ghost. This previously was found under /ghost/debug/, but now is located under
/ghost/settings/labs/. Simply click export and you are all set!</p>
<p>Something I’d like to see in a future release (heck it might exist now), is a
way to automate this. I’d like to just have a cronjob setup to backup my blog
every week (day, hour, minute, whatever, depending on how often you write new
content - not often if you’re me - and how paranoid you are).</p>
<p>The last thing to backup is themes. Casper is the default theme, and lives in
your ghost directory under /<code class="highlighter-rouge">content/themes/</code>. They’ve been updating Casper
fairly frequently. And they suggest just getting rid of the old versions and
installing the latest. I’ve modified the first version I used though, and the
design influenced other parts of my website, so I like to keep old versions. If
you simply move <code class="highlighter-rouge">/content/themes/casper/</code> to something like
<code class="highlighter-rouge">/content/themes/casper_0.5.2/</code> then you’ll see the previous versions when you
go to change your themes in the admin panel.</p>
<h1 id="upgrading">Upgrading</h1>
<p>Now that we’ve backed up our data let’s get to the actual upgrade, shall we?</p>
<ul>
<li>We’ll need to download the latest release, which I find most convenient to
get from their <a href="https://github.com/TryGhost/Ghost/releases">github</a>.</li>
</ul>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">wget https://github.com/TryGhost/Ghost/releases/download/0.5.6/Ghost-0.5.6.zip</code></pre></figure>
<ul>
<li>
<p>I usually make a <code class="highlighter-rouge">tmp</code> directory in <code class="highlighter-rouge">/var/www/ghost/</code>, where I have ghost
installed, and extract this <code class="highlighter-rouge">.zip</code> file there.</p>
</li>
<li>
<p>Next, in <code class="highlighter-rouge">/var/www/ghost/</code> we’ll delete <code class="highlighter-rouge">index.js</code>, and <code class="highlighter-rouge">package.json</code>, then
copy the new ones in.</p>
</li>
</ul>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">sudo rm </span>index.js package.json
<span class="nb">sudo cp</span> ./tmp/index.js ./tmp.package.json .</code></pre></figure>
<ul>
<li>We’ll do the same with <code class="highlighter-rouge">core/</code>.</li>
</ul>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">sudo rm</span> <span class="nt">-rf</span> core/
<span class="nb">sudo cp</span> <span class="nt">-r</span> ./tmp/core/ ./</code></pre></figure>
<ul>
<li>Then we’ll copy over the new Casper version.</li>
</ul>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">sudo mv</span> ./content/themes/casper ./content/themes/casper_0.0.0
<span class="nb">sudo cp</span> <span class="nt">-r</span> ./tmp/content/themes/casper ./content/themes/</code></pre></figure>
<p>You’ll want to replace the <code class="highlighter-rouge">0.0.0</code> with a version number most likely. You can
also just delete the old version if you’d like.</p>
<ul>
<li>Finally, we need to run <code class="highlighter-rouge">sudo npm install --production</code>. This will take a few
minutes, go grab some coffee.</li>
</ul>
<h1 id="testing-and-running-with-forever">Testing and Running with forever</h1>
<p>When it’s done I like to test things out with a <code class="highlighter-rouge">sudo npm start --production</code>.
This will start Ghost. To keep ghost running I had initially used
<a href="http://support.ghost.org/deploying-ghost/#making-ghost-run-forever">forever</a>.
If you don’t have forever installed, run <code class="highlighter-rouge">npm install -g forever</code> within your
ghost directory. I like to then start Ghost with forever with the following
options.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">sudo </span><span class="nv">NODE_ENV</span><span class="o">=</span>production forever <span class="nt">-l</span> /var/www/ghost/forever.log start index.js</code></pre></figure>
<p>The <code class="highlighter-rouge">NODE_ENV</code> definition should be changed to suite your needs, but I’m
running the production environment. If you too are in an environment that isn’t
the development one, and you forget this bit, you’ll probably panic when you
see your blog come up empty. It startled me a bit until I realized my error.
Finally, the <code class="highlighter-rouge">-l /var/www/ghost/forever.log</code> logs the output, which is always
useful.</p>
<h1 id="wrapping-up">Wrapping Up</h1>
<p>We’ve successfully upgraded Ghost! If everything went smoothly you’ll be on the
latest version. So quit reading this and go write a new post! Before you do
though, let me just remind you to clean up a bit, mostly just get rid of that
<code class="highlighter-rouge">./tmp/</code> directory.</p>
<p>Happy ghosting!</p>Brian Koopmanbrian.koopman@yale.eduA perhaps somewhat dated guide on backing up and upgrading a self-hosted instance of Ghost.