Damion in the Cloud

Enterprise Edition

r/Sysadmin - "...Waiting for XYZ routine to complete."

A well explained and objective position from someone non-technical, and a clear response from a system administrator. An example of two commonly interacting yet unique positions, creating a informative dialogue - might make a good interview question.

"I'm a developer that's dealing with new security issues. Recently our computer policy was changed, which has changed functionality of our development computers. Our compile times have had a six fold increase. There are other effects such as downloads and file manipulation taking much longer. Anything that has I/O or network operations are taking much longer.

Our system policy is different than what I'm used to, we have full admin access for each machine and while we use a service (symantec) to push licensed software we are allowed to install or modify our machines however we want. I used procmon to see what was happening during these compile times and noticed a few processes that had a low footprint baseline when I wasn't doing much.

When I wanted to uncompress a file or perform a build I'd see some of the company installed applications or monitoring tools increase up to 25 percent (symantec, lsass, some other security process). I'm not as familiar with lsass, but I spoke with a few developers that said if I'm not performing certificate authentication it shouldn't be doing this. Network access, and file access for both of these were between 100 and 1000 times as many events during the same period for the admin installed applications. It's my understanding that having system software to analyze items during build/ file creating can cause additional issues. EX: anti-virus checking files and causing I/O exceptions due to attempts to access a locked file.

If a developer were to come to you with these issues, specifically build times this long are incredibly terrible for productivity what would you say. In the past I've gone to our IT department about internet speeds and I've been informed that there's no fix (10 people sharing a 10MB/s line through a switch). I know a few other senior devs have been incredibly frustrated with these changes. I understand that there's a chance these are purposeful slowdowns to increase security. But last week I waited 5 hours to perform a ~15gb file transfer from a company external drive and unzip it. Other devs have noted that it seems like every single file creation or change is being monitored."

-"Non-sysadmin; is this a common practice?", /u/TopicStrong

"It does sound like your company's AV is doing real time inspection of every file and operation you're performing. While great from a security standpoint, this is one of those cases where it's shutting down your productivity.

I worked at at a place that bought $BigAVProduct and rolled it out without sufficient testing and department notification and buy in. there was no notification of the new AV, no method or procedure for reporting issues that the AV was causing, no troubleshooting work had ever been attempted. This was in a 300+ person company that has over 50 developers on staff.

The devs complained and the sysadmins fired back that they "must be doing something wrong" because "we never had issues running it on our own IT machines". Every reported problem was brushed off as "those devs and their sketchy tools installed on their machines".

For a variety of reasons I left that company but that was one of the big ones. Throwing arbitrary security measures in place and clicking "Apply" without proper testing or follow-up with affected users.

Here's what I would advise in your situation: Run a specific operation/compile/script that you can time the operation time of. make sure it isn't coded like trash and that it's reasonably well optimized, then record the runtime. Request permission from IT to temporarily disable your AV for performance testing. If they refuse "because security" then offer to disconnect it from the network and to allow IT to be present when you do it. If they still refuse, bring your manager in on the issue and insist. This will do one of two things:

  1. If they agree, it will Give you the performance data to make your case that this AV is unacceptably affecting Dev productivity and to begin troubleshooting with IT and possibly the AV vendor
  2. If they still refuse, it will reveal their hand which is that they know it's causing issues and they don't care or won't fix it or don't know HOW to fix it. In any case, you then have what you need to escalate higher than even IT.

Either way, talk about wasted time , wasted $$$ spent waiting for XYZ routine to complete. Make it real and explain it in terms of the pocketbook."

-Top Response, /u/jacobjkeyes