Troubleshooting - Top Tips to Make Files Work


Bruce Devlin TV-Bay Magazine
Read ezine online
by Bruce Devlin
Issue 110 - February 2016

Troubleshooting - Top Tips to Make Files Work

So, you built yourself the world's most perfect, IT-based file workflow and you have a state-of-the-art media facility? Congratulations, have a beer, but at some point it's going to go wrong!

Given that fewer than 50% of support cases in any company that I have worked at result in changes to software, it helps to have a good analytical approach to finding the causes of problems and communicating them effectively to your suppliers.

What do you do? How do you find those problems?

The first approach is to separate symptoms from causes, and being specific is crucial. Telling support: Our MAM system triggers an API refresh cycle at 2 minutes past every hour, which seems to happen 5 minutes before the storage anomaly is a great starting point. This phrase informs support that you are seeing something happen regularly and that there is some correlation between the symptom and another event in the system / workflow.

As we all know, correlation does not imply causation. So how do you find causes and not symptoms?

One of my favourite troubleshooting techniques for the IT and the broadcast & media industry is The 5 Whys. This technique is widely attributed to one of my all-time engineering heroes Sakichi Toyoda. Wikipedia has the following definition: The 5 Whys is an iterative interrogative technique used to explore the cause-and-effect relationships underlying a particular problem. The primary goal of the technique is to determine the root cause of a defect or problem by repeating the question ˜Why?'

Here is an example where chaining the question why:

The Problem: The media file can't be transcoded

Why #1 The source file can't be opened.

Why #2 Permission failure. The file cannot be opened from the transcode machine with the account associated with the transcode service.

Why #3 Group settings. We check the account on the transcode machine and discover it is in the wrong group so we change it. It still won't open.

Why #4 Drive mount settings. The way in which the drive was mounted in the operating system had an override for the group control. Updating the mount instruction remaps the group to the correct value and the system works. But why did the system go from working to not working?

Why #5 System maintenance. Updating the security of the infrastructure is always risky. You can test 99% of the use cases, but unless you knew to look for the fact that a single machine in a farm was hand-installed in a rush and had the wrong mount overrides then you'll never find the cause of the symptoms until something else changes. Remember that we see symptoms and report them to support, we rarely directly see the causes.

Let's look now at some other questions you can ask yourself when troubleshooting your file-based workflow¦

Is the issue repeatable? Being able to reproduce the symptoms greatly helps to isolate, identify and resolve an issue.

Resolution dependent? Let's try to switch between SD, HD and UHD at different frame rates “ do the symptoms still show? Try to look at the dependency of each parameter of the input.

Is the issue environmental-related? Does the failure occur when there are more people on the system? At a particular time of day? On a specific day of the week? When the playout servers are being updated? Only on specific servers? Only on AWS and OpenShift, but not on Azure?

Time dependent? Does it only fail at a particular time of the day?

Conclusion I won't pretend that finding the causes of any issue in a big software system is easy. First and foremost, it's crucial to differentiate between symptoms and causes. It requires dedication and a methodical nature as well as extreme calm when all around you are panicking. Just remember that at the end of the day, the support person on the end of the telephone is a human who is trying to help find the causes that are hidden within the symptoms that you are seeing. Be nice to them!


Tags: iss110 | dalet | class | academy | sakichi toyoda | openshoft | azure | Bruce Devlin
Contributing Author Bruce Devlin

Read this article in the tv-bay digital magazine
Article Copyright tv-bay limited. All trademarks recognised.
Reproduction of the content strictly prohibited without written consent.

Related Interviews
  • DALET ACADEMY NAB 2015

    DALET ACADEMY NAB 2015

  • Dalet at NAB 2016

    Dalet at NAB 2016

  • Dalet at IBC 2015

    Dalet at IBC 2015

  • DALET SportsPack at NAB 2015

    DALET SportsPack at NAB 2015

  • Dalet at IBC 2014

    Dalet at IBC 2014

  • Dalet at NAB 2014

    Dalet at NAB 2014

  • Dalet at IBC 2013

    Dalet at IBC 2013

  • Dalet at NAB 2013

    Dalet at NAB 2013

  • Dalet at NAB 2012

    Dalet at NAB 2012

  • Classic tubes reinvented by Kino Flo at IBC 2018

    Classic tubes reinvented by Kino Flo at IBC 2018

  • Winner of the LP54 Miller Classic

    Winner of the LP54 Miller Classic


Related Shows
  • BVE 2013 Day 2

    BVE 2013 Day 2


Articles
Increasing Diversity in the UK Indie Market
Sam Addo

London director Sam Addo tells us more about how he aims to change the status quo when it comes to onscreen diversity with his new feature, Cards on the Table.

“Over the past thirty years, filmmakers have made movies telling a wide range of stories, however, in my humble opinion, those with disabilities, from the LGBTQ+ community and or those from ethnic minorities remain underrepresented on screen. For example, it is rare to find a film where the lead character has a disabil ity, and that underrepresentation is worrying.”

Tags: iss139 | cards on the table | feature | diversity | blackmagic | ursa mini pro 4.6k | lut | davinci resolve | Sam Addo
Contributing Author Sam Addo Click to read or download PDF
The Making of Zero
Keith and David Lynch

The Brothers Lynch explain how they created the sinister atmospheric world for their new sci-fi short

In a post-apocalyptic world where humankind has emerged victorious in a war against artificial intelligent machines, a young girl dares to venture into the unknown. This is Zero, the new sci-fi short film from acclaimed British writer-director duo The Brothers Lynch which has premiered at the Tribeca Film Festival.

Tags: iss139 | blackmagic design | davinci resolve | editing | grading | grade | mk2 zeiss | superspeeds | cinema 4d | molinaire | Keith and David Lynch
Contributing Author Keith and David Lynch Click to read or download PDF
Keeping Pace with the Content Revolution
Kevin Fitzgerald These are uniquely challenging times for broadcasters and their technical teams. Not only are they having to negotiate the move to IP-based infrastructures and the introduction of new formats and techniques such as 4K and HDR, they are also having to generate more content than ever before to support OTT and web services as well as traditional linear broadcast.
Tags: iss139 | streamstar | streaming | case 800 | ipx | ipx-3g | Kevin Fitzgerald
Contributing Author Kevin Fitzgerald Click to read or download PDF
State of the Nation - Getting Connected
Dick Hobbs - new We are all familiar with statistics about the growth of the internet. Cisco’s latest report, for instance, says that global IP traffic is increasing at 26% a year, and will reach 4.8 zetabytes a year by 2022. The number of connected devices will be three times the world’s population by the same date.
Tags: iss139 | cisco | kth | clickclean | ibc | Dick Hobbs - new
Contributing Author Dick Hobbs - new Click to read or download PDF
Original KVM or KVM over IP
Jochen Bauer Will the technology used in broadcasting solely consist of IP devices? For years, IP has been entering all areas of life. Especially control room applications as they are typically deployed in broadcasting benefit from the IP revolution in many ways. But an “IP-only broadcast world” is not yet here. Nevertheless, the trend clearly moves towards IP transmission, even though a large part of content production still uses traditional transmission paths. And therefore we continue to live in a hybrid world, using both original and IP-based technology. KVM experts Guntermann und Drunck still rely on both original KVM and KVM-over-IP™ to be able to offer their customers the best of both worlds.
Tags: iss139 | kvm | gdsys | guntermann and drunck | kvm-over-ip | Jochen Bauer
Contributing Author Jochen Bauer Click to read or download PDF