Back to Blog

    DevOps Infrastructure as Code: Tools, Patterns and Careers

    November 11, 2026
    Infrastructure as Code
    DevOps IaC
    Terraform state locking
    Blockchain node deployment
    Featured image for article: DevOps Infrastructure as Code: Tools, Patterns and Careers

    You're staring at a job description that lists Terraform, Pulumi, Ansible, Kubernetes and policy as code, and you're wondering which parts an interviewer can test. Listing tools won't answer that. Hiring managers probe whether you can move a change safely from a pull request to production, recover when it breaks, and prove the result got better. That is the test of DevOps infrastructure as code, and it's learnable if you practise the workflow, not just the syntax.

    What Infrastructure as Code Replaces and Why It Matters

    A validator fails at 3 a.m. and an engineer rebuilds it from memory. The disk size looks right, the firewall mostly matches, but a monitoring agent and one network rule are missing, so the replacement behaves differently from the original. Manual setup works like cooking from memory. You might get close, but you won't get the same dish twice.

    Infrastructure as code is the recipe card. You describe servers, networks, load balancers, DNS records and IAM roles in files, store them in Git, and apply them with a tool that builds exactly what's written. If you need the same node in another region, you run the recipe again instead of reconstructing it by hand.

    A diagram illustrating how Infrastructure as Code replaces manual setup with a repeatable, reliable, and fast process.

    Provisioning Versus Configuration

    IaC covers two jobs that beginners often blur together. Provisioning creates the resources themselves, such as a virtual machine, a managed database or a private network. Configuration decides what runs on those resources, including node software, users, packages and service files.

    Understanding that split helps you choose the right tool later and answer ownership questions in interviews. It also connects to how the machines themselves work, so understanding virtual machines is worth reviewing if hypervisors and guest operating systems still feel fuzzy.

    Where Drift Creeps In

    Configuration drift happens when someone hand edits a live server and the environment slowly stops matching its documented state. The next rebuild misses the edit, or the next automated apply reverses it. Adoption alone doesn't prevent this. In 2025, 89% of respondents reported adopting infrastructure as code, but only 6% reported complete coverage of their cloud infrastructure, according to Firefly's State of IaC report.

    If a change isn't in the repository, nobody can review it, reproduce it or prove it happened.

    Place your own team on that curve. Which resources are still created by hand, and which production fixes exist only in someone's memory or a chat thread? That inventory is your starting backlog, and it's also strong interview material because it shows you can spot unmanaged risk.

    Terraform, Pulumi and Ansible Compared

    Picking a tool means choosing how you describe change and where you record what already exists. The comparison below covers the ownership and state questions interviewers expect you to explain without notes.

    Tool Model and language State handling Best fit
    Terraform Declarative, HashiCorp Configuration Language (HCL) State file, remote backend with locking Provisioning cloud networks, clusters and nodes across providers
    Pulumi Imperative authoring with declarative results; TypeScript, Python, Go or C# State in Pulumi Cloud or a self-managed backend Teams that want application-language tests and shared libraries
    Ansible Procedural playbooks in YAML, idempotent modules No state file; runs against live hosts Installing and configuring software on servers and nodes

    When Each Tool Earns Its Place

    Terraform declares the end state you want and works out how to reach it. It has the broadest provider ecosystem across clouds, which matters when one team manages AWS networking, a Kubernetes cluster and DNS from the same repository. Its state file must live in a remote backend with locking. Since 2023, Terraform has been source available, with OpenTofu as the community fork. If you're learning one provisioning tool first, I'd start here.

    Pulumi lets you write infrastructure in TypeScript, Python, Go or C#, so loops, conditionals, unit tests and shared libraries work the same way they do in application code. That suits teams with strong software engineers who don't want a separate configuration language. The trade-off is discipline. General purpose languages make it easier to hide complex logic inside infrastructure code that reviewers struggle to follow.

    Ansible configures machines over SSH without installing an agent. Its modules are idempotent, meaning running the same playbook twice shouldn't change a correctly configured host the second time. It keeps no state file of its own. That suits installing node software, hardening operating systems and rolling out configuration files, but makes it a weaker choice for tracking cloud resources over time.

    The Combined Pattern and the Hiring Signal

    A common production pattern combines them. Terraform or Pulumi creates the cloud resources, then Ansible configures the software on top.

    A tool list alone is weak evidence. A stronger answer defines the ownership boundary: Terraform or Pulumi creates the network, cluster and nodes; Ansible installs and hardens the software running there. It also names where each result is recorded. Terraform and Pulumi keep infrastructure state so later changes can be planned against existing resources. Ansible checks and changes the live host without maintaining that separate inventory.

    Interviewers can test that distinction directly. They might ask which tool owns a resource, where its state lives, what happens when the same code runs twice and which tool should repair a failed change. You can prove practical knowledge by describing what you provisioned, what you configured, where state was stored and which manual changes disappeared because the tools took over. If you can't answer those points cleanly, your tool knowledge will read as tutorial deep.

    How an IaC Change Moves from Pull Request to Production

    A safe IaC change follows the same path every time, no matter who writes it. The sequence below is the workflow I'd expect any mid level candidate to walk through on a whiteboard.

    A four-step infographic illustrating the workflow for implementing infrastructure as code changes from pull request to production.

    The Seven Steps

    1. Branch and edit. Change the module or configuration in a pull request scoped to one environment. Small scope keeps the review readable and limits what breaks if you're wrong. The common slip is bundling a network change, a node upgrade and an IAM edit into one unreviewable diff.
    2. Format and scan. Run a format check and static scans, for example Checkov or tfsec style policy checks, on the pull request. These catch public storage, open ports and missing encryption before a human spends time reviewing. Skipping scans because the change looks small is how insecure defaults reach production.
    3. Plan in CI. Run terraform plan or pulumi preview in CI and attach the diff to the pull request. The plan is the contract reviewers approve, not the code alone. Running plan only on your laptop means nobody else sees the full impact.
    4. Review the diff. A reviewer approves the plan, paying closest attention to deletions and replaced resources, because those cause outages and data loss. A one line change that forces a database replacement is more dangerous than a hundred lines that add a tag. Rubber stamping plans you haven't read defeats the whole control.
    5. Test the module. Before apply, run plan against a throwaway non production environment. A configuration can parse correctly yet still fail at apply time or misbehave under real access controls. Testing behaviour, not just syntax, is what separates a pipeline from a spell checker.
    6. Merge and apply from the pipeline. Merge to the main branch, then let a protected pipeline run apply with credentials humans can't read directly. The tool acquires a state lock first, and if the lock is held it stops rather than writing over another run. Applying from a laptop bypasses review, logging and credential controls in one move.
    7. Check health after apply. Run post apply checks such as node sync status or RPC endpoint responses. A successful apply only proves the cloud accepted your requests. It doesn't prove the service works.

    Why Locking and Saved Plans Matter

    When applying a configuration, Terraform first secures the workspace state, creates or receives an approved execution plan, executes the resource changes, updates the state snapshot, and then releases the state, as HashiCorp's apply tutorial describes. If a lock already exists, Terraform reports an error and exits rather than proceeding. That refusal is a feature, because two writers updating the same state can corrupt your record of what exists.

    Practical rule: Plan in review, apply only from the pipeline.

    Candidates who can explain why a saved plan supports controlled promotion between environments, and why the lock exists at all, signal production experience. Candidates who only recite commands usually haven't been paged for a state problem yet.

    Recovering from Stale Locks, Failed Applies and Rollbacks

    The failure interviewers ask about most is a stale lock. A CI job gets cancelled mid run and the state stays locked, blocking every later change. Before touching anything, read the lock metadata for the owner and operation, then confirm in the pipeline log that no process still runs.

    Only then should you force release. HashiCorp warns that force releasing can create a multiple writer condition if another process still holds the lock, in its Terraform state locking documentation, so verify the owner and lock ID and record who approved the release. Treating -lock=false as a shortcut is how teams corrupt state while trying to save ten minutes.

    Failed Applies and Honest Rollbacks

    For a failed or partial apply, don't hand edit the state file. Run plan again from the same commit, read the new diff, and let the tool reconcile the resources it already created. Manual state surgery without a fresh plan usually creates a bigger mystery than the original failure.

    Rollback needs equal honesty. Reverting the merge commit restores the old configuration, but deleted databases and disks don't come back. Revert only when the plan shows the old resources can be recreated safely, otherwise roll forward with a fix, and snapshot stateful resources before any replacement.

    Add one more gate while you're here. A plan step that flags new large instances or unbounded storage gives reviewers a cost figure before merge, not after the invoice arrives.

    Deployment Patterns for Blockchain Nodes and Services

    Blockchain infrastructure punishes sloppy boundaries. A bad apply shouldn't cross regions, signing keys shouldn't sit in state, and testnet shouldn't teach you lessons that mainnet pays for. The diagram below shows the regional separation behind the first pattern.

    A diagram illustrating infrastructure as code for deploying blockchain RPC nodes across two cloud regions.

    Three Worked Examples

    RPC node cluster across two regions. Define one reusable module containing the compute, load balancer, firewall rules and monitoring, then instantiate it once per region with a separate state file for each. A bad change in one region can't take down the other, and each region can be rebuilt independently. Teams that would rather buy this capacity than operate it can compare managed options such as Solana Dedicated Nodes from Solana Tracker against the cost of running their own fleet.

    Validator node. Let the infrastructure code manage the virtual machine, disks, firewall rules and monitoring, while signing keys stay in a hardware or managed key store outside the state. The state boundary ends at the machine. Anything that can sign blocks or move funds belongs under separate ownership with its own access reviews.

    Indexer and its database. Use Terraform for the managed database and networking, and Ansible or a Kubernetes manifest for software versions and service configuration. The database state changes slowly and needs careful replacement rules. The indexer code changes often and needs fast rollouts, so forcing both through the same mechanism creates friction in one direction or risk in the other.

    The video below walks through node deployment thinking in practice, which is useful preparation before you try to explain these boundaries out loud.

    Patterns Worth Reusing

    • One module per component. RPC, validator and indexer modules stay independently testable and replaceable.
    • Environment folders with shared modules. Staging, testnet and mainnet differ by inputs, not by copied code.
    • Version pinned providers. Upgrades become reviewed changes instead of surprises during an unrelated apply.
    • Immutable image builds. Rebuild the machine image rather than patching long lived hosts by hand.

    Each pattern prevents a named risk. Separate state limits blast radius, external keys limit compromise, shared modules stop staging and mainnet from drifting apart without anyone noticing, and pinned versions stop a routine change from becoming an unplanned upgrade.

    Drift, Security and Governance Controls to Put in Place

    The most common governance failure starts with good intentions. An engineer fixes an outage through the cloud console at night and never updates the code, so the next apply reverses the fix or the environment diverges without anyone noticing. IaC alone doesn't give you governance. Enforcement does.

    That gap is wider than many admit. A 2025 report covered by The New Stack on governance as code found that fewer than one third of respondents actively monitor and remediate drift, while 17% have no formal drift detection process and 53% rely on reactive or ad hoc checks. If your drift process is hoping someone remembers, you don't have one.

    Risk Control Where it runs
    Manual console change (ClickOps) Scheduled drift detection and break-glass console access Nightly plan against live resources
    Insecure configuration merged Policy-as-code rules for ports, encryption and public access Pull request checks before merge
    Plaintext secret in code or state Secrets manager references and secret scanning Pre-commit hook and CI pipeline
    Validator key compromise Keys kept outside IaC state with separate ownership Key management system and access reviews

    Making Controls Enforceable

    Run scheduled drift detection that plans against live resources and alerts the owning team when reality and code disagree. Limit console access to break glass roles that alert on use, and require a follow up pull request within 24 hours for any emergency change. Without that deadline, emergency fixes become permanent undocumented architecture.

    Add policy as code gates that block public storage, open management ports and unencrypted disks before merge. If you're setting this up for the first time, a walkthrough on how to implement policy as code from Vision helps you structure the rules, and our guide on how to write secure code covers the habits that keep those rules from being bypassed.

    Handle secrets with the same strictness. Store references to a secrets manager, never plaintext values in state or variables, and rotate any credential that appears in a log. For blockchain systems, keep validator keys and state changing access under a separate ownership boundary from the platform team that manages networking, so one compromised account can't reach both.

    What Hiring Managers Test in IaC Candidates

    The skill gap is your opportunity. Survey evidence reported by TechDay on fragmented DevOps indicates that 64% of organizations still use manual steps to deploy infrastructure code and 38% of build pipelines lack security scan gates. A candidate who can close that gap stands out immediately.

    A list of five essential skills for Infrastructure as Code candidates including Terraform, workflows, security, and debugging.

    Four Question Groups to Prepare

    • State and locking. Why does a lock matter, and how do you clear a stale one without corrupting state?
    • Plan review. What does a dangerous diff look like, such as a deletion or a replaced database, and what do you check before approving?
    • Pipeline design. Where do format checks, scans and tests sit, and why do they run on every change instead of as a late manual review?
    • Security. How do you block insecure merges and prove which change altered production?

    Working through sample questions while preparing for practical DevOps interviews from nexus IT group helps you rehearse out loud, and our overview of the DevOps infrastructure team shows how these skills divide across real roles. Blockchain Jobs lists 6,934 open roles across crypto, DeFi and Web3, with Engineering the largest category at 1,624 roles, followed by Legal & Compliance at 551, so there's a wide market for candidates who prepare properly.

    Answer With Metrics, Not Tool Lists

    Structure your strongest answer in three parts: baseline metric, intervention, result. The four DORA metrics are deployment frequency, lead time for changes, change failure rate and time to restore service, and the Atlassian DORA metrics framework identifies an Elite threshold of 0 to 15% for change failure rate and restoration within one hour.

    Bring one before-and-after example from your own work. Senior answers name trade-offs, describe blast radius, and explain how a rollback would work. Junior answers list tools and hope that counts as experience.

    Choose Your First IaC Step and Your Next Role

    Start where the risk is low. If you're new to IaC, pick Terraform or Pulumi, provision one non-production network with remote locked state, and push it through a pull request with a plan review. That exercise covers most of what interviewers probe, and it leaves you with a real artefact to walk through when they ask about state, review and rollback. You'll make mistakes where they're cheap.

    If you already run IaC, measure one DORA metric before and after adding policy checks, then write the result up as career evidence. A measured outcome convinces in a way a tool list never does. If you're hiring, turn the four question groups into a structured scorecard and ask every candidate the same questions, so you're comparing answers rather than personalities.

    Job seekers can browse DevOps and infrastructure roles across Web3 on Blockchain Jobs, using its efficient search and category-tagged listings to find remote-friendly openings that match the tools you've practised. Email alerts send new blockchain and Web3 jobs once a day as they are posted. Your one action this week is simple: open a pull request that adds a plan review step to an existing repository.