[Micronet] docker-machine vs. UCB AWS Direct Connect

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

[Micronet] docker-machine vs. UCB AWS Direct Connect

Greg Merritt
Dear Cloudy Micronetters,

I've hit a nasty catch-22 using docker-machine to set up an AWS EC2 that has a private UCB Direct Connect address. It goes something like this:

The EC2 requires a route to the public internet so you'll be able to do apt-get (and other from-the-Internet downloads) to build out Docker containers.

Meanwhile, AWS EC2 network interfaces on a UCB RFC1918 Direct Connect VPC are given routing tables that (appropriately) only reference campus network ranges, and do not support routing out to the public Internet through campus.

So, how to let your UCB RFC1918-addressed EC2 find the Internet? The most handy way is to also give it a public IP address, which comes with "igw" (Internet gateway) routing.(*) BAM! The EC2 has a pathway to the Internet.

Now, when your EC2 comes up, docker-machine tries to do log in to the EC2 to do its magic. It says, "Hey, EC2 API! What's the IP address of my shiny new EC2 instance?" When it does that, it learns the EC2's public IP address, and it then tries to ssh into the EC2 through that public IP address so that it can continue setup of the shiny new EC2 instance.

Now, if you're doing all of this from campus, then your local desktop (from which you're running docker-machine) awaits a response from the EC2 from the EC2's public interface. But guess what? Your local campus desktop is on campus...and the EC2's routing tables tell it to answer your desktop's ssh conversation via its UCB RFC1918-addressed 10.x.x.x Direct Connect interface.

So...your desktop's docker-machine ssh gets a response from some strange, other IP, and it never completes the conversation.

Another way of saying it: docker-machine knocks on the front door of the EC2, but the EC2 answers at the back door...leaving docker-machine hangin' out in the cold on the front porch.

By the way: if I do *not* let the EC2 have a public IP address -- it only has a back door, so to speak -- then  docker-machine
can ssh just fine! Bummer is that when the setup continues, there is no pathway to the Internet to do apt-get and all the other juicy setup stuff.

...and, in a freaky twist, I can run docker-machine from off campus to fire up such an EC2 with a UCB RFC1918-addressed 10.x.x.x Direct Connect, and it all comes up great, since the campus RFC1918 Direct Connect interface is never involved in the setup conversations. Go figure. Unfortunately, there's no way to readily mange this EC2 from campus after it's up...the ssh key generated by the docker-machine setup process is only good for the public interface, so you have the same front door / back door problem.

Any nice workarounds for this...?

Thanks!

-Greg

(*) A less-handy way would be to fire up an AWS NAT service instance for your EC2's subnet...and that may actually be the realistic solution...but....!


 
-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.

ANNOUNCEMENTS: To send announcements to the Micronet list, please use the [hidden email] list.
Reply | Threaded
Open this post in threaded view
|

Re: [Micronet] docker-machine vs. UCB AWS Direct Connect

Steve Chan

   Can you build your container images on a campus network, and then push the container images to a campus docker registry and just pull the imagine from your docker registry to run in the EC2 hosted docker host?

Steve Chan
[hidden email]
Manager - Enterprise Integration Services, IST-API
SIS Project Integration/Interfaces Team
2111 Bancroft Way office 502A
Berkeley, CA 94720-4990

On Thu, Feb 4, 2016 at 3:55 PM, Greg MERRITT <[hidden email]> wrote:
Dear Cloudy Micronetters,

I've hit a nasty catch-22 using docker-machine to set up an AWS EC2 that has a private UCB Direct Connect address. It goes something like this:

The EC2 requires a route to the public internet so you'll be able to do apt-get (and other from-the-Internet downloads) to build out Docker containers.

Meanwhile, AWS EC2 network interfaces on a UCB RFC1918 Direct Connect VPC are given routing tables that (appropriately) only reference campus network ranges, and do not support routing out to the public Internet through campus.

So, how to let your UCB RFC1918-addressed EC2 find the Internet? The most handy way is to also give it a public IP address, which comes with "igw" (Internet gateway) routing.(*) BAM! The EC2 has a pathway to the Internet.

Now, when your EC2 comes up, docker-machine tries to do log in to the EC2 to do its magic. It says, "Hey, EC2 API! What's the IP address of my shiny new EC2 instance?" When it does that, it learns the EC2's public IP address, and it then tries to ssh into the EC2 through that public IP address so that it can continue setup of the shiny new EC2 instance.

Now, if you're doing all of this from campus, then your local desktop (from which you're running docker-machine) awaits a response from the EC2 from the EC2's public interface. But guess what? Your local campus desktop is on campus...and the EC2's routing tables tell it to answer your desktop's ssh conversation via its UCB RFC1918-addressed 10.x.x.x Direct Connect interface.

So...your desktop's docker-machine ssh gets a response from some strange, other IP, and it never completes the conversation.

Another way of saying it: docker-machine knocks on the front door of the EC2, but the EC2 answers at the back door...leaving docker-machine hangin' out in the cold on the front porch.

By the way: if I do *not* let the EC2 have a public IP address -- it only has a back door, so to speak -- then  docker-machine
can ssh just fine! Bummer is that when the setup continues, there is no pathway to the Internet to do apt-get and all the other juicy setup stuff.

...and, in a freaky twist, I can run docker-machine from off campus to fire up such an EC2 with a UCB RFC1918-addressed 10.x.x.x Direct Connect, and it all comes up great, since the campus RFC1918 Direct Connect interface is never involved in the setup conversations. Go figure. Unfortunately, there's no way to readily mange this EC2 from campus after it's up...the ssh key generated by the docker-machine setup process is only good for the public interface, so you have the same front door / back door problem.

Any nice workarounds for this...?

Thanks!

-Greg

(*) A less-handy way would be to fire up an AWS NAT service instance for your EC2's subnet...and that may actually be the realistic solution...but....!



-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.

ANNOUNCEMENTS: To send announcements to the Micronet list, please use the [hidden email] list.



 
-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.

ANNOUNCEMENTS: To send announcements to the Micronet list, please use the [hidden email] list.
Reply | Threaded
Open this post in threaded view
|

Re: [Micronet] docker-machine vs. UCB AWS Direct Connect

Greg Merritt
In reply to this post by Greg Merritt
Thanks to everyone for the considered replies, both on- and off-list!

Folks tossed out a number of creative ideas for wiring things up a little differently, kind of outside of the Docker tools. At one extreme, one could do Docker deployment differently -- in effect, doing lots of the bits by hand, whether with managing the containers & their deployment, making explicit changes to network config, and so on.

At the other extreme would be a solution that supported the promise of these tools (like docker-machine & docker-compose), namely, that they take care of all of this stuff so that you never have to think about it. ;)

Maybe the ultimate solution is something like a fix to docker-machine that lets it understand the Direct Connect / RFC1918 context, perhaps via a flag when invoked, or something similar.

A lot to chew on...I will digest...

Thanks!

-Greg

 
-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.

ANNOUNCEMENTS: To send announcements to the Micronet list, please use the [hidden email] list.