Systems Enigneer @Charter Communications

Greenwood Village, CO
Aug 2020 - Present

This is where my exposure to Linux systems really started to expand. Once I joined the team at Charter Communications, 100% of my day to day work was with Linux and other open-source projects. Coming from a mostly Microsoft background, I can say that Linux, and the concept of open-source, has really grown on me. I’ve been involved with many projects since my time here, so below I will briefly list a few and what they consisted of:

• Lead engineer and architect for containerization of a proprietary, complex and mission-critical application. The application was running on EOL systems and using heavily outdated application components. Some of the tasks involved the following:
  • Deployment and configuration of a Kubernetes cluster (Calico, eBPF (still in progress...), LINSTOR, HAPorxy, etc...);
  • Setting up observability, monitoring and alerting of the entire cluster (Prometheus, Alertmanager, Loki, Grafana);
  • Decoupling application components into separate K8S workloads;
  • Building custom container images for each application component that would replicate current production environment;
  • Writing custom Helm charts for deployment of multiple K8S resources/workloads for the application;
  • Configuring authentication, authorization and admission control with PoLP in mind;
  • Drafting a roadmap and step-by-step procedure for gradually upgrading each application component;
  • Mimicking this deployment in other data centers for geographical redundancy (still in progress).
• Lead engineer and architect for deployment of a centralized and highly available syslog infrastructure. The syslog servers had to be capable of ingesting millions of syslog messages per second, from over 200,000 network devices spread across the country. Once again, below is a brief list of tasks this project consisted of:
  • At the top of inbound syslog injection were two IPVS load balancers (active/standby);
  • Behind the load balancers were five syslog servers. The syslog solution of choice was “syslog-ng”;
  • Because IPVS is a low-level load balancer, lacking a lot useful features that HAProxy offers, I had to develop a custom Python systemd service that would automate monitoring and adding/removing of available backend syslog servers;
  • All inbound syslog messages were persisted to centralized storage, which was running DRBD;
  • This same solution was deployed at another geographical location for redundancy in standby mode. All syslog messages were continuously synced between those two locations with DRBD.
• Lead engineer and architect for introducing observability, monitoring and alerting to an existing environment hosting multiple mission-critical services.
  • The entire solution was deployed in existing Kubernetes environment with Helm charts developed for our specific scenario;
  • The mission-critical services were either hosted on different VMs, running in an existing VMware or K8S environment;
  • Prometheus was deployed for scraping metrics from all components, from the physical hardware to the applications;
  • Loki was deployed to scrape logs for the same systems and applications;
  • Alertmanager would send alerts via email and a custom developed webhook, written in Node.js, to send alerts to Webex.

Systems Enigneer @Boyd Gaming

Las Vegas, NV
Jul 2018 - Aug 2020

This was my first exposure in a systems role and it’s sort of a long story about how I got the job with practically zero prior experience, but it’s probably the best example of how quickly I was able to adapt and excel in a completely unfamiliar environment. Here’s a brief list of a few projects I worked on:

• Lead developer for automating deployments of Windows Servers. This was sort of a proprietary development of a Cloudforms-like portal for self-service provisioning and was completely written in PowerShell. This involved the following:
  • Windows ADK to develop custom Windows Server images that adhered to the latest security standards, Microsoft updates, Infosec provided EPP agents, and so on. These images were later used as approved gold templates;
  • Developed with concurrency that automated deployment of multiple VMs across multiple vCenters across the country;
  • Automation of guest OS configuration, like networking, domain joining, installation of different Windows Server roles and so on.
• Boyd Gaming has hotels & casinos spread across the country. As an engineer, I was frequently responsible for traveling to different locations and setting them up with infrastructure for on-prem private clouds:
  • Physical installation and cabling of servers, network switches and storage devices;
  • Installation of ESXi hypervisors, vCenters and configuration of DVS, HA, DRS and so on;
  • Deployment of a set of VMs, using the proprietary self-provisioning solution described above, that provided standard services like directory services, ADFS, DNS, WSUS, DFS, and many more.
• There were multiple migration projects during acquisitions of other hotels & casinos. This was a highly coordinated effort that consisted of tasks such as:
  • Migration of AD objects between AD forests and domains using ADMT;
  • Migration of Windows Servers hosting various server roles/services;
  • Complete replacement or upgrade of ESXi hypervisors and vCenters;
  • Virtualization of legacy physical servers (hot or cold P2V);
  • Migration of thousands of VMs between vCenters.

NOC Technician @Switch

Las Vegas, NV
Feb 2017 - Jul 2018

This was my first job in the US and it’s also a very good example of how I started with very little experience and was able to adapt and excel in the role. During my time here I also earned my Cisco CCNA Routing & Switching certification. Although my day consisted mostly of troubleshooting and monitoring tasks, it was an invaluable stepping stone for my later career development.

• Monitoring and root cause analysis in case of customer circuit interruptions. Depending on the type of circuit, this would involve reviewing logs on Switch network devices, working with the customer and other circuit providers.
• Switch’s upstream connectivity was another mission critical component of this role. Monitoring and troubleshooting of upstream BGP connections to various upstream providers such as AT&T, Verizon, CenturyLink, Zayo, Windstream and more. This would frequenlty involve collaboration with upstream providers in order to find the root cause of an interruption.
• I also assisted with activating new customer and upstream circuits. This consisted of working with customers, Switch’s network engineers and engineers from other carriers. In cases where an activation doesn’t go according to plan, it would involve extensive troubleshooting.

MCSA: Windows Server 2012

Issued Feb 2020

CCNA: Routing & Switching

Issued Dec 2017

Below is an incomplete list of technologies I have worked with over the years. I don't claim to be a master with any of these, but all of them were conscientiously researched for the task at hand and I have enough experience to know how and where they can be used. I can’t write extensively about each one in this résumé, but hopefully it will illustrate my broad exposure.