ISP Tracker features, agent or service questions

@DeanONH: We need to be able to download all collected data in the form usable by MS Excel. I suggest comma delimited format. I believe that someone is going to respond to an email that you sent about this. This is important to us as well and recently became a higher priority.

I'll check. Maybe the fix was not sent to production yesterday if they were working on other things.

Hi James, I have a new hardware agent which I received last week and it's failed to remain running for more than about 10 minutes. I plugged it in and created an agent on the website, successfully communicating with it and setting it up. Hardware agents are pre-created and waiting for you to activate it once you receive it. When you install or 'create' software agents, you are prompted to pick the operating system version. The installer eventually prompts you to enter your login credentials so that it can find out which account to assign your new agent to. You mention showing 'continue installation' would be a software agent and unrelated to your hardware agent. When you first create a software agent, it shows in 'Created' status with 'complete installation'. Those are removed 24hrs later so as not to confuse members. Now let's look at the hardware agent. When your hardware agent was prepared and ready to ship, an email containing the word 'Activation' would have been sent to you. That email contains all of the information you need to know how to connect your agent and activate it into your account. I think you are missing this email as it explains how to get it up and running and activated into your account. Without this information, you might end up going down the path you have. Reset does not do anything on the agents, we disable them because people have trashed their agents by pressing buttons without knowing what they mean. Since you see a single solid LED on, it means the agent is running as it should be, it's just waiting to be activated into your account. Let's see if we can take care of that first then we can try to solve what's going on with any software agents you might have created. First, can you post the agent ID of the hardware please. There is a white sticker on the side of the hardware agent, please share that number.

Hi, Let's start with some basic things. First, near the top of the starter_linux_ocp.sh script, is the version number 1.0.3? Second, did you set the 'start' path in the script? It's on the 7th line I believe. start="/xxx" (set the path where you want to have the agent downloaded to and running from). Let's start with that.

Hi James, It turns you the agent ID you gave earlier was in fact a software agent that you had installed and not the hardware agent that you ordered along with extended. Therefore, the update was done on your software agent and not your hardware agent. We see that you activated the hardware agent before putting it online as the instructions explain which actually lead to our understanding that we need to add a little more logic into the activation process to prevent this. Once you connect your new hardware agent, all will be fine and its reports will be in Extended mode. Apologies for any confusion but it lead to something good at least.

Hi, After spending quite a bit of time on this yesterday, we think this is something we have to dig into deeper. We ran some tests, have some results that seem to imply there is something that is not quite right and this means needing to do some longer term testing. What's happening is something like this. The two conditions that can trigger a full speed test are if the agent thinks something has changed or where there is an outage. The agent has a built in algorithm that triggers speed testing when certain conditions arise. For example, it does some ongoing tests to determine if there are any slowdowns or latency from averages that it's built up from the time where you start the service. These conditions aren't outages but we think the agent may be doing the above tests and sending those as outages based speed tests. Where there is an outage, the agent sets itself up to do a speed test as soon as it can reach the Internet again. This means there is going to be a delay between the time that the outage started and when the speed test was able to start and complete. Trying to correlate the two have been challenging because we have to determine if the agent went inactive, disconnected or if there was in fact an outage. By the time the speed test is send to the database and shown in your graph, the times would not align. There is no way to know with 100% certainty which speed test and which outage happened at the same time. The agent could have also seen a short outage or disconnection during that time and triggered another short test or full speed test. We have talked about this in the past and have yet to find a solution that would be 100% accurate. When reports are in Extended mode, you also gain the historical menu which you can look through to correlate some of this data but it's a manual process at the moment. We don't want to automate correlations until we are 110% sure when events are related. The point is, it's something we have planned to write some test code for today so that we can monitor but it'll be a while before we can definitely know more and much data has to be collected. We have put this post in the issue so we know to update it when we have more information. We appreciate your bringing it up and if the above doesn't help, feel free to ask what ever you need and I'll try to simplify it so that you can find more value in your current reports.

Hi, We are constantly working on the services and every now and then, are not aware that something broke. I've passed this on to the developers who tell me this will be fixed in a few minutes. Thank you very much for taking the time to tell us.

Thank you for letting us know that things are better now. Let us know if you need anything else.

Hi, I created your own post so it does not get lost in the other persons post.

Hi, Yes, you are right, this is a bug and is getting fixed right this moment. Thank you very much for letting us know. Feel free to ask anything you need about the service and we'll be happy to help.

it could mean a hardware/signal problem somewhere" So that must extend to any issues effecting an individual residence - right? Yes, correct. It could be the modem/WiFi device at the customer location and/or it could be with the provider. There is no way to know when it comes to signal problems because only the provider controls their own network and they do not share this kind of information. So an outage must then be something that effects multiple customers for a provider. Not always but if you were to eliminate signal levels at your location and still see either no connectivity or outages, then chances are others in your area are also affected. If you were to motivate one or two others in your area to use OutagesIO, then you could more easily confirm these things. For example, if you are watching the map at the time of a problem, you would see other agents in your area also going into an 'Inactive' state. If there was an outage then it is likely that your location and your neighbors would show a matching outage and/or date/time, hop, IP, even the provider name if you are all using the same provider. How does the app know when its an actual outage? The app part are the reports that you're looking at when you log into app.outagesio.com. The agent is what is monitoring the Internet connection. It is purely IP based monitoring. In the simplest description, it maintains a constant ongoing connection to one of our networks using standard port 80. If something becomes inaccessible from source (the agent location) to destination (one of our networks), it immediately starts running a series of IP based tests to confirm that there is a hop down somewhere. When you see Inactive, it does not mean an outage, it is simply a notice that your agent is no longer able to communicate with our network. IF there is an outage and you did not restart the agent, it will send that report once it is able to communicate again. If you keep seeing Inactive but you are not seeing outages, it means the problem is not purely IP based, it could be any number of things. Knowing a bit about the kind of network the provider is using, coax, DSL, Wisp can help to narrow the possibilities. Getting others in your area to also monitor can give a much clearer picture of what is going on in your area. If I want to see when I have outages even if they effect just me do I need to either watch the heartbeat for blips or better yet turn on the email alerts to know when the heartbeat issues happen? Is there a better way? The heartbeat does nothing but let you know the agent is communicating. We just wanted something easy to visualize so that members could clearly tell when their agent was communicating. The heartbeat simply means that the agent is communicating every 15/20 seconds or so. You cannot determine anything by the heartbeat other than the agent is currently communicating or not. In the basic reports, the options are to receive an email for any Inactive condition. It is just a way to get your attention so that you can come to the dashboard and see what is going on. Again, if you see inactives without outages, you are likely not experiencing outages. Having said that, I am talking about the majority of consumers that have fairly standard setup, not multiple switches, multiple routers and other things that could throw the algorithm off. In most cases, it's pretty accurate. In your case,it seems that the provider controls your modem, changing its private IP fairly regularly. Maybe that is indicative of some problems in your area too. I hope this helps you, it's a lot of information :).

Hi, We are in a transition adding some of the business side features into OutagesIO. This may help explain what is going on: Introducing Extended reports We are also trying to update all of the help notes in the pages where you configure etc. Grayed out items mean they are not available in the current level of reports meaning Basic but are available in Extended reports. Some of the things you see are in transition where we need to catch up with help here and there. We do post in the new features section of this forum however to let members know what we're up to. It's s work in progress. We also mentioned in a few places that we're happy to give members a preview, all we ask for is some feedback in the forums on how OutagesIO has or is helping. A problem with our free service is that we have a lot of people using it but next to no one giving us feedback which makes it hard to know about problems, ideas, suggestions and more importantly, how it's helping.

Hi, I'm afraid there is not at this time. It took a lot of effort and cost to get the code working right and to become what it is. We have talked about the possibility of making it available to some developers at some point or offering an API but we've not had much time to get past the thinking stage yet. In fact, we've removed code from it to make it as small and basic as possible, putting more emphasis on the network side rather than into the agent side. What do you have in mind? You can PM me if you prefer.

Yeah, its already on the way. Unfortunately we don't really have any other contacts in the neighbourhood to make that work, so I'll have to make do with using a different router.

Hello Alex I reset the software in the taskbar and waited a day. There was no change in what appeared in the dashboard, so I ininstalled the software and reinstalled it. Setup Failed. 0x80070643 Fatal error during installation. This happened after I entered my user name and password that I use to open up the dashboard site. There was no other email listing a user name and password in the Agent 31518 Notification email. Regards, John

Unfortunately, no. We ended up factory resetting them so that we could reload our firmware. Maybe some sort of DHCP/fixed IP mix up where the agent/s could no longer get a DHCP IP since they were expecting a static DHCP assignment. Really not sure. All I can tell you is that here also, they would not connect to our DHCP server, at all. I really do not know what happened but very happy that we were able to reload them and get them working. Please let us know how things are going after you use them for a while.

Great question. The main point of the article is that there is no way to get anything conclusive. It's just another test, another stat that needs to be taken into account with more information. Speed testing is definitely useful but is a moment in time test using shared bandwidth where the test can go across networks we have no control over and have no idea how they are set up. More importantly, most of these tests seem to terminate on CDNs which are optimized to cache data mainly for streaming and other media. That's why we used a big file so that the transfer could settle and we could (should) get a fairly sustained speed going. Even when using iperf which is a more real world test, it is inconclusive because the test travels over networks we don't control. The article is mainly questions like why did the transfer drop down to kilobytes when it's a 50Mpbs pipe that is barely being used? I don't think we are implying anything about the provider but pointing out that we simply don't know since we don't control their network. We can make some assumptions by testing different kinds of data to different locations but in the end, it is inconclusive. Yes, a web site could be slower but 30 seconds is way up there in terms of time to render pages. The point there was simply to see if there was something going on with the file transfer specifically or the overall bandwidth. Meaning, maybe the sustained speed caught the attention of an application manager which automatically tried to keep the transfer at a certain speed. No idea. One of the more interesting things during the testing was that we checked the hops to the destination and could see that the provider actually had something on the edge of the data center we were testing to. Meaning, our location, the provider, level3, back to the provider then into the DC. It seemed odd that we could never get anything close to 50Mbps. I'm getting about 2-300Mbps on my wired speedtest, it's still useful in that i will spot a relative degradation but i'm pretty sure if we used iperf or qperf to test between our connections we'd almost always get 1000Mbps. When you say test between your connections, do you mean with the same provider or would those tests go across multiple network owners to get from source to destination. Testing between servers in a data center or even to another data center managed by the same org usually always shows the correct throughput but that's because in these cases, the network owner is under obligation to make those speeds can handle all of the traffic required. Cable companies don't work this way, it's a best effort service that has 'acceptable' ranges to cover all their needs. I cannot speak with authority since I've never been part of a cable company but I've spent years fighting with them to get what we pay for and to get them to fix problems for us and customers when we offered ISP and MSP services. We did a lot of this kind of testing and while speeds can remain around what ever they tested at, most of the time, it's up and down as expected. My guess is the speedtest uses a very small file and so on my big connection it's measuring the time taken to establish the connection too which is why it nets out at about a quarter actually bandwidth? In fact, OTM tests against fast.com which has an interest in making sure that consumers are getting the bandwidth they are supposed to be getting. If you are seeing what you think is a set up time as a delay, it's possible since it has to spawn a process, hit at least three servers, calculate the results, etc. The browser test is instant if you go to fast.com. However, hardware agents that we sell do something different, similar to what you said. They run a shorter, smaller test to get a quick average. Unlike the usual speed testing that people are doing, we aren't really interested in fully saturating everyone's connections over and over again. The main purpose of our speed test is not about finding out what your maximum speed is but if you have usable bandwidth for your requirements. Our speed testing tools are still in their infancy as we try to find the best way to give the most useful information without fully saturating and using up all kinds of data since many are on data plans. The speed testing will end up being to and from our network only at some point when we can find the best way to get useful results. We've also tested limiting the speed to fast.com so as not to be using up users bandwidth/data or wonderful fast.com service. BTW, there are a lot of interesting blogs/articles about fast.com and why they built the test and made it download only. it's quite interesting. I hope this helps clarify the bit of testing we did that day :).

Hi Peachy, Internally, we call this classifying the networks. For typical setups, this determination is quite accurate considering that both the local network and the providers often use private IPs. Because of the above, there is no way to be one hundred percent sure that the software got it right and in not so typical setups, human intervention is required. The user must know their own network in order to understand the results. In fact, this is something mentioned in these forums quite a bit, that in some cases, the agent cannot determine the local network from the providers and the user has to know this. I'll give you an example. Lets say a private network with multiple routers, gateways and no static. The algorithm might think that the first gateway it sees is yours because the next one is say four hops away. It cannot reliably determine where your network ends and where the providers starts. Lets say you have a couple of wireless routers being used as hot spots (APs) and in turns, those use your main router as their gateway. Then your providers IP is another private IP. The code may not be able to determine where your network ends and where the providers starts. In some cases, providers control everything from their plant to the Ethernet port or access point they provide at the customers home. All these devices are on the providers network and the customer gateway is inside the providers network. The code may show that the gateway is where the local network ends. While software can do a lot, in non typical setups, it is assumed that the user knows their own environment and can take this into consideration. There is something on the table about allowing the user to 'adjust' their networks so the algorithm is aware of each network. For example, the user could specify which hops are in their own network and the algorithm would simply know this when it tries to classify the networks. The classification code will get better as we get feedback and read about how members are using the service. We do have tests happening even as I type this and as we confirm something works, are able to constantly replicate a good result, then we look at adding it into production. There are a few things being tested in terms of the classification, it just takes time to prove or disprove it. Great question.