Transcript
Caroline:
Welcome to our webcast. My name is Caroline Lim, and I'm the Marketing Programs Manager here at Infochimps. I'm excited about our Ironfan community webcast today. Here at Infochimps we're seeing lots of new development on Ironfan.
Before we jump into that, a little housekeeping. First, we are recording this webcast. We'll send a link to the video via email early next week, so watch your inbox. Secondly, ask questions. We want your participation in this webcast. In your GoToWebinar control panel you can see a hand button, which is the hand raising functionality. Click on that button to raise your hand and we'll call on you, at which time you can ask your question to Nathan directly.
We'll be monitoring this channel and we'll ask your questions during the webcast. We will also be hosting a Q&A session towards the end of the webcast. Without further ado, let me introduce you to our DevOps engineer Nathan Eliot.
Nathan:
Howdy, guys. This is Nathan Eliot. I'm actually going to kick this off with a poll. I'd like to get a feel for where everybody is in their familiarity with Ironfan so far, so Caroline's going to launch the poll now, and I think that will come up on your GoToWebinar. We'll give you all a moment to answer while we're …
Caroline:
Okay. The votes are coming in. We'll give it another minute for everyone to put in their votes. If you haven't already, please place your vote. We'll go ahead and close the votes.
Nathan:
Looks like we've got a fair amount of somewhat familiar and then scattering of the rest. For the not very familiar people, I apologize in advance. I am going to go kind of fast. Ironfan is our tool for running large clusters. It's currently only able to deploy on AWS, but we've got a bunch of stuff in the pipeline for other large clouds.
It takes a DSL written in Ruby and transforms that into a list of expected machines and then goes out to the various cloud providers and gets them in sync with everything. It uses Chef for a large amount of its underpinnings, both for the native functionality of running Chef cookbooks on the server to make it the server that you want rather than just the bare server. Also, it's kind of an essential store of information that's communicated between the launching computer and the target computer.
The core of Ironfan is the Knife plugin, and the homebase stores all of your personal information that you need to connect to your cloud accounts. The plugin itself is what connects Knife into the rest of the architecture and does all of the heavy lifting.
Unfortunately, I could quickly pass through a couple of those components. Here we've got a [inaudible 03:59] cookbook and, excuse me. I'm on someone else's computer reaching through to my own, so I'm not sure on the key combos. Here we have various clusters that we use in our testing environment for various sets of development. You'll see the fairly simple layout of the language. That's more or less it.
There are some other components under the hood, like Silverware, which does some nice coordinated magic to make it very easy to find the system that is providing a particular service from another system, and from there it's largely just the coordination of various components of that along with using [Git] to keep things in coordination with the rest of your [team]. That's pretty much the basics. Caroline, I think, had another poll.
Caroline:
We're going to launch the second poll right now. Please place your vote and we'll give everyone a couple of seconds. I'll give it just another moment for everyone to, please, place your votes. We'll go ahead and close it. Here you go, Nathan.
Nathan:
It seems like most of you have not actually used other cloud orchestration tools before, which I'm impressed that you're coming to Ironfan first. Thank you. And I'll remind you, if you have any questions, you can raise your hand and Caroline will flag me and let me know that the question has come in and that you can ask it either by chat or verbally once she's given you the green light there.
In the meantime, what's new in Ironfan, we are currently landing continuous integration, which our aim is to have the entirety of our Ironfan tool set and the entirety of our Chef cookbooks and related information synchronized across all our structures in a well tested manner. I can maybe talk a bit more about that later if there's interest. Other things that we've landed, we've recently landed a [Ubuntu] precise bootstrap with which you can burn Ubuntu precise images, which are in a number of ways more reliable than the previous natty image, which wasn't bad, but had some quirks.
What other picture [inaudible 07:49]? We have a number of things under the hood that are improving stability across all of our cookbooks, including better testing of the core image cookbooks that only get laid down once, and some stuff in the works includes a provider plugin for vSphere, which we are working on internally and in collaboration with some of their existing Serengeti code, which was based on Ironfan and with several people who have interest in vSphere implementation.
We've also had some discussion on the email list about an Open Stack implementation and the continuous integration will be over the next year, advancing into continuous deployment of architecture, which, again, I can discuss at more length if people are interested. I think that's pretty much the bulk of what's new and what's coming, so I think at this point we're going to start opening up for general Q&A. If you've got a question, can you go ahead and flag Caroline so that she can get you on the line.
Caroline:
We already have a few questions in the queue. First one's from Ryan. "I heard you guys were working on continuous integration. What's the current status?"
Nathan:
The current status is I am probably a week from landing the initial implementation of CI. It uses Jenkins for its core and works by pulling in all of the changes that have been in-queued in the testing branch of the pantry, standing up an instance that has all of the cookbooks that we want tested on it, which then runs to completion. We watch for a special cookbook called Ironcuke, which goes through the Silverware announcements, which are the way that we tell other systems that we provide a particular capability, and it verifies that those announcements are, in fact, true.
That, like I said, will be landing in some light form next week and we will be hardening up over the coming weeks, and then further on the horizon we've got continuous deployments, which takes further manifests about your specific architecture and stands up testing instances of specific instances that you already have to ensure that changes in cookbooks don't destroy your existing production before you [push drip]. That's it in a nutshell.
Caroline:
Thank you, Nathan. The next question is from Gregory. "How can I get support for Ironfan?"
Nathan:
There are a couple of places to get support. If it's an easily described issue with existing functionality, you can always open an issue on our Github page. That's incorrect. Hold on. My apologies, folks. I do not see that until just now. Our Github page for Ironfan has an issues queue and we keep a close eye on it, and we try to respond quickly to it. There is also the Ironfan user's email list for more general questions and discussion, and we've already got at least a little discussion going around the possibility of new providers. Then if you're interested in a demo of our platform, we have a signup page on our website and finally, I will be doing a talk at South by Southwest Interactive, which will go into more detail about how you can develop inside [inaudible 12:47].
Caroline:
Next question is from [Dika]. "You guys say Ironfan speeds up development. How?"
Nathan:
I think actually one of the largest things that Ironfan allows you to do that you can't with traditional architecture is to give up and start over without losing your place. We have an internal concept of recapitulation, which means that any of our servers could, if we backed up the data, be destroyed entirely and recreated usually in less than ten minutes. For some of the more complicated things it can take up to 20 minutes for a whole suite of servers to come back up, but we have also got a number of things in place such as EBS-backed volumes that live beyond the lifetime of the cluster itself, unless otherwise specified, that allow you to destroy and recreate a cluster in a cleaner newer state.
And that has turned out to be a short circuit for a number of operational problems because instead of having to figure out how to back out of a bad situation, you just have to figure out how to migrate away. That also frees up the developers to be more bold in their own changes because we can always give them a test cluster that's very close to what production is going to look like and nobody cares about the contents of, besides the developers. If they set it on fire, we kill it and we bring it back anew.
Caroline:
The next question is from [Shawnette]. "Why is Ironfan better than other cloud orchestration tools?"
Nathan:
Modesty makes me hesitate at better, but one of the nicest things about Ironfan is how lightweight it is. Compared to a lot of other orchestration tools Ironfan has very low requirements. It's Ruby and a bunch of gems and a couple of accounts on various providers. Right now we use Opscode for our hosted Chef and we use AWS for our infrastructure as a service provider and it's that and Ruby 1.9 and a bunch of gems, which are all installed by bundlers. The speed with which you can get started is way higher than some other tools, and going forward it's going to be a lot less bound to the infrastructure as a service provider in that there will be other providers that give similar functionality and allow you to build the same thing across AWS, Rackspace, HP Cloud and other implementations as we get them going.
Caroline:
This is another question from Ryan. "What other Open Source projects do you do?"
Nathan:
Personally I'm at the fringes of this. We've got, I think our best is Wukong. It's a data transformation tool that can run in a number of different contexts. It can be run locally to test and develop. It can be run in a Hadoop context for large scale MapReduce. It can be run in a storm [inaudible 16:30] context for streaming data, but we're working on other contexts like, I believe there's one in development for Web interaction, so an API could be hit and run that code when it's hit. We've got Wukong. What else do we have [inaudible 16:56]?
We've got Wonderdog for interconnecting ElasticSearch in Hadoop, and we've got our lunchlady which is the way that we order lunch around the office, and I'm reminded that I've not put in my order today, so I hope that somebody remembered for me. We've got a whole lot more on the Infochimps community page, [inaudible 17:30] does a nice job of making configuration simple. Vaya Con Dios, is the LED connector to Wukong that I was talking about. Rubix is a Ruby client for Zabix, which is our monitoring system. That's currently in our enterprise edition only, but I am pushing and we'll see if it gets put into Open Source.
We have a couple of others. Ironcuke, is the core of the testing that I was talking about earlier in continuous integration and Cube is a system developed by Square that we've built some cookbooks around, and then Gorrillib is our library of Ruby convenience methods that Ironfan is developed heavily upon.
Caroline:
All right. The next question is from Logan. "If I want to try out Ironfan, what's the best way to get started?"
Nathan:
It's an intense process, but the first step would be to go to that Ironfan page. It has some documentation on how to get started. What you'd be looking to do is make a clone of the Ironfan homebase, which you will find links for there. In your own personal git repo, you'd want to clone the example credentials and rename them as well as your clone at the homebase, and then you'd want to start populating that with the various bits of information that are going to tie your command line on your local computer to your accounts on AWS and hosted Chef.
There are a number of steps we're trying everyday to make them a little bit easier and there should be good instructions, but if the instructions confuse you, please come to the mailing list and let us know. We want to get our documentation more cleaned up and ready than it is, and that's the best way that we can know about it.
Caroline:
The next question is from Ryan. "Will you have future webinars where we can actually see how you use Ironfan at the command lines and how you do your CI, for example? I feel the documentation on Github isn't always completely clear as to set it up locally."
Nathan:
I can walk you through some of that now, actually because we've got a good bit of time. Bear with me as I open up the various bits that you will need. Test monkey is our test environment. It's how we get all our cookbooks cleaned up and working without endangering any of our other setups. The core of the setup is going to be in your knife folder. You've got a symlink from credentials that should be pointed at your actual credentials repo, which you will want to submodule in.
In that credentials repo, you will want to put a key matching your Chef, your Opscode username. If your Opscode username is different from your username on your command line, you will have to do a little bit of initial setup in your .bashrc, or whatever your command line resource script is, to set the Chef user to your Chef username and then it will pick up the correct key and use that to get to your Chef account. You will also need to open up your knife-org, and in there you will find things like the Opscode Chef organization, the AWS account ID that you'll need to set, and then sometimes you will also need to have, in fact, currently you will also need to have a Knife user file in which you set your AWS access keys so that you can do things in the AWS cloud.
You may also need, if your AWS username is different from your Chef and command line username, there is another invocation. I believe it's documented. I can't remember it off the top of my head, but it is something on the order of Chef config, hold on. Let me see if there's an example in here. [inaudible 23:06] account ID. Extra keys. Cluster. SSH user. No. I'm sorry. That's the user for the eventual machine. I am not finding this at the moment. If it is not documented, I will make sure it is documented shortly, but for now know that there is a trick, and if you are having problems because of the difference in usernames, please post on the list and I will speed up the documentation of that.
Once you have all of those bits in place you can start doing Knife cluster commands. Knife cluster commands are currently wrapped by bundle exec. You can and probably sometimes should wrap them further in quotation. At some point, and this will be announced on the mailing list, I will subsume the bundle so that it comes in on every Knife cluster command so that you don't have to do all this wrapping yourself, but for now, Knife cluster lists and -f says show the facets as well. This goes through all of the cluster files that you have locally and lifts out where they are and what facet they create.
This is a really useful thing, when you have a lot of clusters, for remembering what goes where. El Ridiculoso is our testing cluster. It has a number of things in it including just a very small system called El Ridiculoso Albano, which is the villager because our AMI burning cluster is called Burninator and the system that burns it itself is [inaudible 25:18], so we had to have a village as well.
I'm going to start using my shortcuts. KC is just a shortcut that produces the bundle exec Knife cluster. I'm going to show El Ridiculoso, and this will go out to the Chef and AWS servers and come back with a compiled list of what all is available. Also comparing against the local DSL to see what things you could start. I'm going to just start Albano, but I could, with the same command, start the whole of the cluster by just removing the facet specification, or if these were facets that had multiple servers, I could indicate just one server with that implication.
I'm missing the launch. Knife cluster launch, El Ridiculoso Albano, it first loads up the various resources and then synchronizes them for the entire cluster including security groups and the access between security groups, and then it starts up the actual machines, creates the cloud machine manifest here, and now I think we have to wait while some API calls go on, unfortunately. This is the slowest portion of Ironfan, which is the talking with external APIs. The responsiveness is sometimes a little frustrating.
While it's doing that, I'm going to try switching back and showing you some more of what's going on on the cluster side. We have the El Ridiculoso cluster. We start by declaring some stuff about the [inaudible 27:51] cloud that we are in. We've declared the size of the machine we want and under the hood, Ironfan is finding a number of bits of information about that machine and using it in its calculations so that resources on the machine are more available. We've chosen an AMI image name, which is from a lift that we have in our Knife setup. See if I can get you the list real quick to show you what it looks like.
These AMIs are not publicly available yet. We need to do some vetting before we would be comfortable doing so, but you can see the general format here it lists information about the machine requested and then some specifics about the AMI ID, the user that you SSH in as and the strip that was used to bootstrap. Switch back. The machine has launched. We have tagged the instance in AWS so that we can find it again on future shows with its cluster facet index and name. We've tagged its EBS root volume because all of our instances are EBS backed for stability. That way you can stop and start them without loss of data.
We've set its termination flag false, but we could set it true if we set a permanent flag in that cloud EC2 statement that we have earlier, put a statement of permanent true here. Then the next time that we sync the flag that prevents accidental termination of an instance would be set in AWS and you would have to manually go and unset the permanent flag and resync in order to be able to stop it. That's an extra safeguard that keeps very important servers from falling over. We don't actually need it here though.
Then after it's set back it synchronizes the EBS volumes and launches the actual machine waiting for it to be available by SSH and then doing some final cleanup tasks about the machine. Then if we go and show it again, we will get a much more populated list of information about what's currently running and what isn't, and we can also SSH onto the machine. This may still take some time to come back correctly because the step that deploys user accounts has already run, so I have a working machine that I've just logged into for the first time.
So receiving some extra warnings, but we have the IP address. We have a whole lot of other things, and if we look we can even see where it's at in its Chef client [inaudible 31:53] that's running in the background as an initial service on the 12.04 image. It only runs once as an initial service and then exits, which we find to be better for stability of the Chef client if it's run periodically but not run as a long standing process. At this point, are there further questions?
Caroline:
Because Ryan did prompt the initial question with the amazing work session that we just went through, he has a follow-up question. "What approaches would you recommend to test cookbooks locally? Virtual boxes are the question of the moment in V4, so what can we do in the interim?"
Nathan:
That's not actually where we tend to focus. We've set up an automated testing environment that does depend on our AWS accounts, however, I would say you could probably get a lot with Food Critic, just for local venting, and if you're really strongly interested in it, again, hop on the users group and see who else is interested in a local BM implementation because we've got some internal interest and hearing some external interest, especially in helping to develop that implementation would push it up the chain.
Caroline:
We actually have our first hand raising. We will put Jim on the line right now. I am currently unmuting you, Jim, so you will be able to talk directly to Nathan.
Jim:
I don't know if I should talk. I'm at home, but the question is [inaudible 33:41] has a lot of the features that Ironfan has but Java [centric] and their road map is quite a bit ahead, I guess. They have some [inaudible 33:55] scaling features and stuff like that, but how would you see that progress, your road map versus their road map?
Nathan:
To be quite honest, I don't watch the competition as much as I ought, so I'm going to go look at their feature list, [inaudible 34:15] spelled it correctly.
Jim:
The thing is [clearly for] integrating with Chef, so you could take a Chef recipe now and they wrap it up. They're using Groovy to write recipes.
Nathan:
I think I actually did look at these guys a little. They are much more of a platform as a service focus, which is perfectly fine. We're focused more on the infrastructure, where they're the ones that do [Bosch], correct?
Jim:
No. Actually Bosch is the VMware guys. It's the stuff that, I can't remember. It's based on [inaudible 35:15]
Nathan:
Yes. I am getting them confused. In that case, I can't speak as much to what they are doing now except to say that what I'm seeing so far looks to be the kind of thing that we actually tend to avoid in some of our development, in that it is, for the most part, looks like configuration rather than code. One of the things we've found is that having code in every part of the stack makes solving certain problems a lot easier. We tend to bump up against problems where what we're dealing with is a configuration because we can't as easily and as frequently inject intelligence into configuration. You can always inject more intelligence into code. Sometimes that's not a great idea, and we do have to keep some parsimony about where we put the code for a particular solution, but having the options available seems to make it a lot more flexible.
That said, I think [Clarify] is much more focused on their product than we are as a company, as a whole because part of what we do is big data and we're also focused on deploying full stack integrations that involve a number of other moving parts like large elastic search clusters which require [tuning], large storm configurations that fan out and connect with a large number of different end points. Without looking too much further, I believe we're not too terribly far behind them, but have very different focuses, and it looks like they are advertising a little more vendor neutrality than we have at the moment, but we are catching up to them quickly.
Caroline:
I hope that answers your question, Jim. Is there anything else you'd like to ask Nathan? Thank you very much. I will go ahead and mute you for the rest of the evening. The next question is from [Chaz]. "What's going on with the cookbook repos on GitHub? Have they settled into one monolithic repo for good?"
Nathan:
At the moment we're settled on a monolithic repo for a number of reasons. The largest was that we found we needed the monolithic repo on our end for doing the kind of development that crosses a number of cookbooks. It clearly becomes painful when it's one repo for cookbook to keep them all downloaded and [edit] on them. We've also found that keeping that in sync with a sub [treed] repo per cookbook setup was, at the time, problematic. Now that we've got some more automation in place, we might revisit that again, especially since it seemed to be the way that Chef has gone, but for the moment, we're settled on monolithic, and we've done a couple of modifications to various tools that we use in order to adapt unto that, and we'll continue to do so until we find strong compelling arguments to go the other way again.
Caroline:
We have someone else who raised their hand. Ryan, I will go ahead and unmute you, so please ask your question to Nathan directly.
Ryan:
Hi, Nathan. It's Ryan here. I was wondering if you have any capability of showing us how you do your CI at the moment, at all. Like a demonstration.
Nathan:
It's a little clumpy, but I can certainly show you the core of it with the caveat that this is all subject to change as we evolve the infrastructure. I'll have to open this internal. This is our Jenkins server and, like I said, some of this is subject to change, most notably that we are going to be going to a nightly build rather than a triggered build. At the moment, any change that is pushed into the testing branch on either of our pantries there, Ironfan Pantry or Ironfan Enterprise, will trigger a call back from Github to Jenkins that will launch these control jobs which are largely just control jobs, they don't have any internal tests themselves, but they will in turn launch the Ironfan CI job.
I'm going to show you a job that succeeded rather than the failed job, just for the sake of my own ego. Here it holds in the testmonkey homebase that I was working with earlier. Pulls in the most recent revision and starts doing the steps necessary to get a fully working environment. Actually, I'm glad that you asked this because I'm noticing one bit that I forgot on the earlier question of how do you get your environment going. One important step is running Bundle Install, which it does here. I'm sorry. It's actually running Bundle Update here. I did fix that. On initial run it will run Bundle Install on [inaudible 41:37] update. It synchronizes changes up to the Chef server. At the moment we don't have good change detection on rolls because they are packaged with the home base rather than the cookbooks for a number of reasons I can explain if somebody needs me to.
We then go out to the pantries and get all of the cookbooks that are listed in the berkfile. Install those locally from the testing branch, and then we do it again for the staging branch because we need to be able to tell what differences have occurred. We look through all of the cookbooks, find those that have versions that have been enqueued, or changes that have been enqueued, and in that case, it is five of our cookbooks [inaudible 42:35] cluster of Jenkins route53, Silverware and Zookeeper. We upload those to the Chef server, and I apologize for the formatting on this. There's not much I can do, but what we do next is a bunch of Knife cluster commands, or lift Knife cluster show and then a Knife cluster launch on the Villager, and then we wait while the Villager stands up and watch for the completion.
I actually will show you the failure as well because that's also important, but once we've completed, then we tear down El Ridiculoso and we notify the upstream projects, and then, keep on hitting that key combo and that's the wrong one. Then from there, once the Ironfan CI is kicked off to completion, it will trigger the stage which goes and takes those changes that have passed inspection and moves them over to the staging branch, and then we'll additionally call out to our environments and go out to our home bases, and I'm making this distinction because it will go into the home base and deploy a Chef environment with all of those cookbooks pegged to the staging version.
What you have is a continuous flow of code from your developers into the testing branch, through the gate in here in CI and then into not just the staging branch, but also your stage Chef server. I will also show you quickly the failure mode for the CI, which is down here at the bottom. I apologize.
This is Ironcuke, and Ironcuke, if it fails, spits out a list of all of the failures that it has. Some of these are false positives, in fact, I'm pretty sure all of these are false positives because we're still fixing up some of the things that make Ironcuke compatible with the rest of the environment, but it gives you a fairly nicely formatted list of what it expected and what it actually got. And if Ironcuke isn't the reason it failed, you will see the stack dump for whatever did fail, which is pretty useful for easily finding and fixing the problem that caused the failure. That's where we are pretty much at the moment. Like I said, the [fan out] is the next thing and then some slight reorganization so that it only does the in queuing once a night and the stand up of a testing server once a night rather than any time somebody pushes.
Caroline:
Thank you, Ryan, for raising your hand. I will go ahead and mute you now. I invite anyone else who would like to raise their hand to go ahead and do so. We appreciate the community engagement. In the meantime, a question from Joe. "I heard about Ironfan in the context of big data. Is it usable for other use cases as well?"
Nathan:
Yes. I actually hope that people will come to me with those use cases as they encounter them, but I came from a Web development background and one of the things that drew me to Infochimps in the first place was the vision of what their tool set could do even in that context of very easily standing up a large number of coordinated services and standing up more as you need them, almost. We don't yet have an auto-scaling implementation, but we're edging ever closer to the place where one could be easily done.
Caroline:
The next question is from Cliff. "How does Infochimps leverage Ironfan?"
Nathan:
We use it everywhere. I have, in fact, been pretty aggressive about removing parts of our architecture that weren't built in Ironfan or were built in earlier less capable versions. Everyone of our developers has Ironfan access. They use it in their day to day routine. All of them contribute cookbook changes. All of them contribute, not all of them contribute on the core plugin, but a number do. The only thing that we don't build in Ironfan are things that we think we could but which somebody else is better at building, like Github is better at building git repositories with nice front ends, so there we don't, but everywhere else we do.
Caroline:
Another question from Ryan. "How fast are you able to spin up a Hadoop cluster?"
Nathan:
We're still at the 20 minute mark. We did a presentation which I don't think [inaudible 47:59] slide [inaudible 48:00] right here, unfortunately, but if you go search Slideshare for Ironfan you will find me and Dhruv Bansal, at a Chef conference last year standing up a Hadoop cluster in 20 minutes and a demo of an unfortunately old stand up of the same type of cluster internal MBS. Like I said, we're not there with the VMs in Version 4, but we are hankering for it and we will definitely be going there again in the future.
Caroline:
This next question is from [Lorena]. "While researching on Ironfan and big data, I came across an Apache implementation called WHIRR, which behaves similar to that of Ironfan. Can you explain regarding this name?"
Nathan:
Again, I haven't been paying as much attention to my competition as I ought, so let me go look now. Yes, it does seem to be similar in use case. Let's look at their quick start guide. It looks like WHIRR may be largely focused around deploying just the Apache apps, or at least that seems to be a large portion of its primary focus. They're discussing configuring and deploying and running a Hadoop job in the quick start guide, which kind of implies that their focus is on the Hadoop stack alone. We do Hadoop stack. We use Cloudera's packages of it and we do a lot of custom tuning on top of it, but that is not, by any means, the total of what we do, and I don't know whether Whirr can handle stuff outside of its environment.
Caroline: The next question is from Chaz. "Last I researched Service Discovery in Ironfan is [inaudible 50:25]. Has Zookeeper been considered?"
Nathan:
That's an old backing. Currently Service Discovery is actually backed in the Chef server because it's already a dependency of Ironfan for a number of reasons and it fits just fine. Since Service Discovery is all about finding what node provides a particular thing, we put the announcements for that node into the Chef node, not sure what to call that, Chef node bucket, I guess. We put it into the node attributes themselves and then search those node attributes via the Chef search API, which seems to be fairly performant and not requiring a whole lot of extra work on our part.
As things go forward, and as our need for announcements starts to increase beyond the capabilities of announcing only during a Chef run, we'll probably look at more backing and I'm not sure. We're probably going to go with something like Zookeeper though because we like the ability to easily scale it and we have a number of instances of scaled Zookeeper clusters already to work from.
Caroline:
We have time for a couple more questions. If you have any questions, please feel free to put them in the question box. This is another question from Cliff. "Are we hiring, Nathan?"
Nathan:
I believe we are. Let me see if I can pull up our jobs page real quick. We're always hiring for good engineers, is the short answer. I'm not sure exactly which ones we're hungry for at the moment, but we hire more about the person than about the position, so if you've got serious chops, doesn't matter if they're Ruby chops. It could be, but when I started here I knew next to nothing about Ruby. The important part is can you work self directed? Are you in love with the kind of work you do? Can you solve big hard problems? Because we've got a lot of them. Yes, we're looking for sales engineers, data engineers, and ops engineers at the moment. We'll probably be looking for the foreseeable future.
Caroline:
Okay. Does anyone else have any questions? Last call for questions.
Nathan:
Thank you, everybody for coming.
