What Are “DevOps” Skills?

/in , /by

Someone recently asked me about DevOps ‘courses’ which got me thinking about what the required skillsets are for getting into roles that include the word “DevOps” in the title or description.

I’ve been working in various “DevOpsy” roles professionally for about 7 years now and I still don’t know exactly what it means. Each company seems to have a different definition, and the philosophy that it all started from is now only a distant memory :-D.

That said, every “DevOps” role I’ve had has drawn on some combination of the following skillsets:

  • Linux Skills
  • Networking and Web/HTTP skills. And DNS. Always DNS. And caching, oh god the caching problems.
  • Some cloud provider’s stack — AWS, GCP, Azure, whatever — knowing how different architectures are implemented, using the tools they expose to infrastucture designers and operators.
  • Familiarity with a CI/CD process (specific tools are usually not important in interviews, as long as you’re comfortable with ONE of them).
  • Generalized troubleshooting and problem solving skills. Almost every problem you face as a DevOps person will be 15% known, 85% unknown. The ability to quickly learn about the problem domain and start troubleshooting is invaluable.
  • Be comfortable with the software development process — how software gets written and deployed. Know the basics of software tooling — git, the basics of the language your devs are writing in, debugging tools for that language/environment, etc.
  • Be *really* comfortable with reading through (and puzzling over) large codebases.

It *really* helps to have some programming (developing software with a team of other devs) experience, although it’s not a hard requirement.

I’m trying to stay away from specific tools recommendations in this post, but several important ones come to mind.

My question to you: what skills do you find yourself using at your DevOpsified job?

I was wrong about Docker

/in , , , /by

I recently found an answer that I gave someone to a Facebook chat question in ~2016-2017, and was amazed at how “technically right” I was (i.e. totally wrong for the career question that I was answering).

I just wanted to post this to show that I am often too technologically focused, and that this can often get in the way of getting hired and getting things done the way everyone else is doing them. On the flipside, my heart is in the right place and this approach certainly wouldn’t have HURT the person I was giving advice to.

Here it is!

Yeah, I should make some docker vids! So, LXC is the original Linux-native implementation of containers. There was an implementation before, called OpenVZ, but that relied on a patched Linux kernel and I don’t think it’s used much anymore.

The mainline kernel itself exposes the “ingredients” for containers (user/process namespacing, cgroups, etc.) and LXC was one of the original implementations of ‘containers’ using these native building blocks.

Docker was originally a wrapper around LXC, but has now actually created their own implementation of the container back-end (using the same exposed ‘ingredients’ from the Linux kernel that LXC uses).

LXC (and now LXD) is under-hyped, solid, and a bit more intuitive for learning (more like FreeBSD Jails, less hype/magic than Docker). I think it’s good to learn LXC or FreeBSD jails before jumping into Docker.

All the concepts from LXC are totally applicable to Docker as well.

Docker, as a software product, layers some extra tools on top of this basic ‘containerization’ tech: they have their docker ‘hubs’ — pre-built application containers. They have extra networking stuff on top of what you’d get with LXC. They have something that looks like very basic container linking/service discovery. They have enormous amounts of hype, which can help you get a high-paying job (seriously. It’s worth learning just for that.)

I’m torn, because I’ve used Docker since before it was an open-source product (it used to be a paid thing called DotCloud), and have hated it for almost the entire time. It’s a complex, unstable, and questionably architected piece of software. It’s also incredibly overhyped and misused.

BUT: Docker has some great ideas in it, and is absolutely worth learning for your career. As a technology, it leaves a lot to be desired.

I’d love to hear what you think of Docker after getting comfortable with LXC. LXC will teach you the concepts that underlie all of this containerization stuff, and Docker adds some new features (and new headaches) onto that.

Let me know what you think, if/when you start your Docker journey!

System Administration Careers: Frequently Asked Questions

/in /by

If you’re wondering whether or not a System Administration career is right for you, this article might help. Someone just sent me some questions about what a career in System Administration is like, and asked some questions that I hadn’t thought to answer on YouTube or here on the tutorialinux blog. Since I’ve been working in various IT Operations and Software Development disciplines professionally for the last 7 years, I love talking about this stuff. Here are my two cents:

 

What are the Responsibilities a Linux / Unix System Administrator?

Architecting, Building, and Troubleshooting/Maintaining the technical infrastructure for a company or software product. At a high level, you’re responsible for figuring out what the technical requirements are and using existing software/products/tools to provide those requirements (or occasionally writing your own). Early-career focuses on implementing predefined task-chunks:

  • Figure out why this server is doing XYZ
  • Fix a problem with our software deployment automation code
  • Increase storage size for one of the database clusters
  • Replace failed disks at a datacenter
  • Create a web server configuration file for a new project that Dev is working on

Senior-level positions often have more design/architecture:

  • Deep introspection of the OS and the software product you’re supporting
  • Design a new deployment pipeline
  • Build a technical team for a new project
  • Evaluate new software that would change how we run our infrastructure (e.g. containers + scheduler vs. VMs + config management)
  • Troubleshoot tough problems (OS-level issues, bugs arising from a confluence of several edge cases across the stack, etc.)

 

What education/experience/credentials are required to do this job?

Read more

New Books Added to the Sysadmin Reading List!

/in , , /by

I’ve just added some more amazing books to the sysadmin reading list page. These are books that I’ve read over the last year which are so spectacular that I think every sysadmin (or aspiring sysadmin) needs to read and digest them.

Some standouts from the list above:

The Linux Command Line: A Complete Introduction

I watched a project manager with no hands-on technical experience go through this book. After about a week, he was comfortable moving around on the command line, (git) pulling down the latest version of the code that his team was working on, running shell scripts, and doing other basic Linux/Unix tasks. I think it’s fair to say that this book put him on track for becoming a savage beast (for a PM).

 

The Practice of Cloud System Administration: DevOps and SRE Practices for Web Services, Volume 2

This is one of the greatest system administration / infrastructure / DevOps / System Reliability Engineering (SRE) / Platform Engineering books I’ve ever read.

Do you remember how the UNIX and Linux System Administration Handbook, 4th Edition taught you how Unix and Linux work in a deep and practical way, and how it showed you what it takes to administrate a single server or a small group of servers?

That’s what this book does, except for modern large-scale infrastructure (hundreds/thousands of servers). It will show you how to design infrastructure properly. It will teach you how to think about reliability. It will show you how to take (most of) the pain out of monitoring and alerting.

I can’t say enough good things about it. Seriously, if you want to be a Google/Amazon/<insert-enormous-tech-company-here>-level Admin, this book will introduce you to the skills you need.

Buy it, now!

 

 

Okay, I’m ending this post here because I need to get to work — a place where I will undoubtedly use the skills you can learn from these books.

So read these books, and then get out there and start doing this stuff!

Advice to an Aspiring Sysadmin

/in , /by

I get a *lot* of e-mails and YouTube messages asking me how to become a system administrator. Usually I just point people at the blog or YouTube channel, but yesterday I decided to write up something a bit more complete. Here’s a slightly modified version for easy reading.

 

 

Asking Questions, doing Research, and getting Help

This is worth its virtual weight in gold: http://www.catb.org/esr/faqs/smart-questions.html

Subreddits are great. Check out /r/linux, /r/sysadmin, /r/networking, /r/programming, and others. Long story short, find communities that are doing things you’re interested in.

No one will sign up to mentor you, but if you show interest and effort, work on practical projects, and ask well-prepared questions, people will generally treat you well and help you out.

Read more

How to Choose a Career in IT

/in /by

Most aspiring tech people make a critical mistake when evaluating ideas for a career. They approach the problem from a 30,000-foot view, saying to themselves, “I might like a career in finance.” Then, they try to work out a more detailed niche, before making plans for getting there. This can work sometimes, but if you find yourself getting stuck in this high-level thinking, perhaps a more practical approach is right for you.

Read more

How to Get Into a Programming Career

/4 Comments/in , /by

I was just talking to a friend about programming. She’s interested for all kinds of reasons, not the least of which is that she’s read all about how ‘programming is the future’ and that tech people make tons of money.

We talked about some things that I thought would be useful to share more openly — so here it is: my advice for getting into programming, System Administration, or any other technology path.

Read more

The Education of a System Administrator

/4 Comments/in , /by

Here’s something that surprises many people: I actually don’t have a college degree. Before I got into System Administration, I did all kinds of things: I’ve been a soldier, carpet salesman, martial arts teacher, Chinese massage (Tui Na) practitioner, data entry temp, bakery worker, and a few other things.

On the surface, these don’t look like they are related to System Administration, but each of these other attempts at ‘finding the right career’ taught me something valuable that I still use today in System Administration and Development work.

For example, all that ‘unrelated‘ experience has taught me how to

Read more

Where to Find Remote Programming Work

/3 Comments/in , /by

I remember sitting at a system engineering gig a few years ago, fighting an angry LDAP server and talking about my goal of sitting on a beach and doing sysadmin and programming work from my laptop. My manager, a smart and practical fellow, laughed and told me it was a pipe dream and that such work simply didn’t exist. Two years later, I can work from the beach every day, if I feel like it.

Everyone loves remote work — whether it’s system administration, database administration, testing, QA, remote programming work, or something totally different. As with anything else, there are some downsides, but the advantages to both companies and employees are huge. Lower office costs for companies, fewer interruptions for employees; the list goes on and on. It’s still early in the ‘remote work’ timeline, and some businesses still need to get used to the idea. If you’re interested in an exhaustive pro-and-con list in book format, check out Remote: Office Not Required.

Here are a few of the sites I’ve used to search for (and get) both full-time and contracting work:

 

Read more