void vmsHttpFlvTrafficAnalyzer::FindFlvDownloadInInBetweenDlgs(const vmsHttpTrafficCollector *pTraffic, HTTPDLG pWebPage, LPCSTR pszSwfUrl, LPCSTR pszFlashVars, LPCSTR pszSwfContext, HTTPDLGLIST vFlvs, FindFlvDownloadInInBetweenDlgs_Type enFindType)
{
LOGFN ("vmsHttpFlvTrafficAnalyzer::FindFlvDownloadInInBetweenDlgs");
if (!pszSwfUrl)
{
assert (pWebPage != NULL);
if (!pWebPage)
return;
pszSwfUrl = pWebPage->strRequestUrl.c_str ();
}
string strBody;
if (!pszSwfContext)
{
assert (pWebPage != NULL);
if (!pWebPage)
return;
strBody.assign ((LPSTR)&pWebPage->vbResponseBody [0], pWebPage->vbResponseBody.size ());
pszSwfContext = strBody.c_str ();
}
int nStartIndex = pWebPage ? pTraffic->getHttpDialogIndex (pWebPage) + 1 : 0;
HTTPDLGLIST vFlvs2;
DWORD adwCTs [] = {
vmsHttpTrafficCollector::HttpDialog::AMF,
vmsHttpTrafficCollector::HttpDialog::X_WWW_FORM_URL_ENCODED,
vmsHttpTrafficCollector::HttpDialog::XML,
vmsHttpTrafficCollector::HttpDialog::HTML,
};
for (int iCT = 0; iCT < sizeof (adwCTs) / sizeof (DWORD); iCT++)
{
HTTPDLGLIST vBetwDlgs;
pTraffic->GetHttpDialogs (nStartIndex, -1, adwCTs [iCT], vBetwDlgs);
HTTPDLGLIST vBetwDlgs2;
vector <HTTPDLGLIST> vBetwToFlvMap;
for (size_t iDlg = 0; iDlg < vBetwDlgs.size (); iDlg++)
{
HTTPDLG pDlg = vBetwDlgs [iDlg];
if (pDlg->vbResponseBody.empty ())
continue;
if (pDlg->enCT == vmsHttpTrafficCollector::HttpDialog::HTML &&
pDlg->vbResponseBody.size () > 1000)
continue;
string strBody;
if (pDlg->enCT == vmsHttpTrafficCollector::HttpDialog::AMF)
{
for (size_t i = 0; i < pDlg->vbResponseBody.size (); i++)
{
if (pDlg->vbResponseBody [i] >= ' ')
strBody += pDlg->vbResponseBody [i];
}
}
else if (pDlg->enCT == vmsHttpTrafficCollector::HttpDialog::X_WWW_FORM_URL_ENCODED)
{
string str; str.assign ((LPSTR)&pDlg->vbResponseBody [0], pDlg->vbResponseBody.size ());
vmsHttpHelper::DecodeUrl (str.c_str (), strBody);
}
else
{
strBody.assign ((LPSTR)&pDlg->vbResponseBody [0], pDlg->vbResponseBody.size ());
}
if (enFindType == FDT_EXACTURLMATCH)
FindDialogsByUrlExactMatch (strBody.c_str (), vFlvs, vFlvs2);
else if (enFindType == FDT_FILEURLMATCH)
FindDialogsByFileUrlMatch (strBody.c_str (), vFlvs, vFlvs2);
else if (enFindType == FDT_BYUNIQUEWORDS)
FindDialogsByUniqueUrlWordsMatch (strBody.c_str (), vFlvs, vFlvs2);
else
return;
if (!vFlvs2.empty ())
{
vBetwDlgs2.push_back (pDlg);
vBetwToFlvMap.push_back (vFlvs2);
}
}
if (vBetwDlgs2.empty ())
continue;
LOGHTTPDLGLIST ("List of dialogs with FLVs", vBetwDlgs2);
LOGsnl ("Found dialogs with FLVs. Find their source.");
HTTPDLGLIST vBetwDlgsFoundInSwfContext;
FindDialogsByUrlExactMatch (pszSwfContext, vBetwDlgs2, vBetwDlgsFoundInSwfContext);
LOGHTTPDLGLIST ("List of these dialogs belonging to source page (exact match)", vBetwDlgsFoundInSwfContext);
if (vBetwDlgsFoundInSwfContext.empty ())
{
FindDialogsByFileUrlMatch (pszSwfContext, vBetwDlgs, vBetwDlgsFoundInSwfContext);
LOGHTTPDLGLIST ("List of these dialogs belonging to source page (file url match)", vBetwDlgsFoundInSwfContext);
}
if (vBetwDlgsFoundInSwfContext.empty ())
{
FindDialogsByUniqueUrlWordsMatch (pszSwfContext, vBetwDlgs, vBetwDlgsFoundInSwfContext);
LOGHTTPDLGLIST ("List of these dialogs belonging to source page (unique url word match)", vBetwDlgsFoundInSwfContext);
}
if (vBetwDlgsFoundInSwfContext.empty ())
{
FindDialogsByUniqueUrlWordsMatch2 (pszSwfUrl, vBetwDlgs, vBetwDlgsFoundInSwfContext);
LOGHTTPDLGLIST ("List of these dialogs belonging to source page (unique url words2 match)", vBetwDlgsFoundInSwfContext);
}
if (!vBetwDlgsFoundInSwfContext.empty ())
{
wstring wstrTitle;
if (pszFlashVars)
ExtractTitleFromFlashVars (pWebPage, pszFlashVars, wstrTitle);
if (wstrTitle.empty () && pWebPage)
ExtractTitleFromHtml (pWebPage, wstrTitle);
for (size_t i = 0; i < vBetwDlgsFoundInSwfContext.size (); i++)
{
for (size_t j = 0; j < vBetwDlgs2.size (); j++)
{
if (vBetwDlgs2 [j] == vBetwDlgsFoundInSwfContext [i])
{
HTTPDLG pBetwDlg = vBetwDlgs2 [j];
for (size_t k = 0; k < vBetwToFlvMap [j].size (); k++)
{
HTTPDLG pFlvDlg = vBetwToFlvMap [j][k];
wstring wstrTitle2;
if (pBetwDlg->enCT == vmsHttpTrafficCollector::HttpDialog::X_WWW_FORM_URL_ENCODED)
ExtractTitleFromXWwwFormUrlEncoded (pBetwDlg, wstrTitle2);
else if (pBetwDlg->enCT == vmsHttpTrafficCollector::HttpDialog::XML)
ExtractTitleFromXml (pBetwDlg, wstrTitle2, pFlvDlg);
if (wstrTitle2.empty ())
wstrTitle2 = wstrTitle;
AddFlvDownload (pWebPage, pFlvDlg, wstrTitle2.c_str ());
}
break;
}
}
}
return;
}
}
}
HRESULT vmsHttpFlvTrafficAnalyzer2::FindFlvDownloadsInHtmlPage_imp(LPCSTR pszHtmlPageUrl, const vmsHttpTrafficCollector *pTraffic, DWORD dwFlags, int nCallDepth)
{
const vmsHttpTrafficCollector::HttpDialog *pDlg = pTraffic->FindDialogByRequestUrl (pszHtmlPageUrl,
vmsHttpTrafficCollector::HttpDialog::HTML);
if (!pDlg && (dwFlags & FTAFND_DONT_DOWNLOAD) == 0)
{
if (vmsDownloadForTrafficCollector::MakeDownload (pszHtmlPageUrl))
{
pDlg = pTraffic->FindDialogByRequestUrl (pszHtmlPageUrl,
vmsHttpTrafficCollector::HttpDialog::HTML);
}
}
if (!pDlg)
return E_INVALIDARG;
vector <const vmsHttpTrafficCollector::HttpDialog*> vFlvs;
vmsWinSockHttpTrafficAnalyzer::FindOutgoingDialogs (pTraffic,
pTraffic->getHttpDialogIndex (pDlg),
vmsHttpTrafficCollector::HttpDialog::FLV, (dwFlags & FTAFND_DONT_FIND_UNIQUE) != 0, vFlvs);
wstring wstrTitle;
ExtractTitleFromHtml (pDlg, wstrTitle);
for (int i = 0; i < vFlvs.size (); i++)
AddFlvDownload (pDlg, vFlvs [i], wstrTitle.c_str ());
if (vFlvs.empty ())
{
vector <const vmsHttpTrafficCollector::HttpDialog*> vDlgs;
vmsWinSockHttpTrafficAnalyzer::FindOutgoingDialogs (pTraffic,
pTraffic->getHttpDialogIndex (pDlg),
vmsHttpTrafficCollector::HttpDialog::X_WWW_FORM_URL_ENCODED | vmsHttpTrafficCollector::HttpDialog::XML,
(dwFlags & FTAFND_DONT_FIND_UNIQUE) != 0, vDlgs);
for (int i = 0; i < vDlgs.size (); i++)
{
const vmsHttpTrafficCollector::HttpDialog* pDlg2 = vDlgs [i];
vFlvs.clear ();
vmsWinSockHttpTrafficAnalyzer::FindOutgoingDialogs (pTraffic,
pTraffic->getHttpDialogIndex (pDlg2),
vmsHttpTrafficCollector::HttpDialog::FLV,
(dwFlags & FTAFND_DONT_FIND_UNIQUE) != 0, vFlvs);
if (!vFlvs.empty ())
{
for (int j = 0; j < vFlvs.size (); j++)
{
wstring wstrTitle2;
if (pDlg2->enCT == vmsHttpTrafficCollector::HttpDialog::X_WWW_FORM_URL_ENCODED)
ExtractTitleFromXWwwFormUrlEncoded (pDlg2, wstrTitle2);
else
ExtractTitleFromXml (pDlg2, wstrTitle2, vFlvs [j]);
if (wstrTitle2.empty ())
wstrTitle2 = wstrTitle;
AddFlvDownload (pDlg, vFlvs [j], wstrTitle2.c_str ());
}
}
}
}
if (nCallDepth == 0)
if (vFlvs.empty ())
{
LPCSTR pszHtml = (LPCSTR)pDlg->getBodyText ();
vector <string> v1, v2;
vmsHtmlParser::findAllTagsHeads (pszHtml, "iframe", v1);
vmsHtmlParser::findAllTagsHeads (pszHtml, "frame", v2);
for (int iVector = 0; iVector < 2; iVector++)
{
vector <string> &v = iVector ? v2 : v1;
for (size_t i = 0; i < v.size (); i++)
{
vmsHtmlTagParser tag (v [i].c_str ());
const vmsHtmlTagParser::TagAttribute *src = tag.findAttribute ("src");
if (!src || src->strValue.empty ())
continue;
string strHtmlUrl;
vmsHtmlParser::DecodeHtmlSpecEntites (src->strValue.c_str (), strHtmlUrl);
if (strHtmlUrl.empty ())
continue;
FindFlvDownloadsInHtmlPage_imp (strHtmlUrl.c_str (), pTraffic, dwFlags, nCallDepth+1);
}
}
}
return m_vDownloads.empty () ? S_FALSE : S_OK;
}
HRESULT vmsHttpFlvTrafficAnalyzer4::FindFlvDownloads(LPCSTR pszHtmlPageUrl, const vmsHttpTrafficCollector *pTraffic)
{
LOGFN ("vmsHttpFlvTrafficAnalyzer4::FindFlvDownloads");
m_vDownloads.clear ();
const vmsHttpTrafficCollector::HttpDialog* pWebPage =
pTraffic->FindDialogByRequestUrl (pszHtmlPageUrl, vmsHttpTrafficCollector::HttpDialog::HTML);
if (!pWebPage)
return E_INVALIDARG;
if (pWebPage->vbResponseBody.empty ())
return S_FALSE;
string strPageText;
strPageText.assign ((LPSTR)&pWebPage->vbResponseBody [0], pWebPage->vbResponseBody.size ());
int nStartIndex = pTraffic->getHttpDialogIndex (pWebPage) + 1;
HTTPDLGLIST vFlvs;
pTraffic->GetHttpDialogs (nStartIndex, -1,
vmsHttpTrafficCollector::HttpDialog::FLV, vFlvs);
HTTPDLGLIST vFlvs2;
FindDialogsByUrlExactMatch (strPageText.c_str (), vFlvs, vFlvs2);
if (vFlvs2.empty ())
FindDialogsByFileUrlMatch (strPageText.c_str (), vFlvs, vFlvs2);
if (!vFlvs2.empty ())
{
wstring wstrTitle;
ExtractTitleFromHtml (pWebPage, wstrTitle);
for (size_t i = 0; i < vFlvs2.size (); i++)
AddFlvDownload (pWebPage, vFlvs2 [i], wstrTitle.c_str ());
return S_OK;
}
string strBody;
strBody.assign ((LPSTR)&pWebPage->vbResponseBody [0], pWebPage->vbResponseBody.size ());
FindFlvDownloadInInBetweenDlgs (pTraffic, pWebPage, NULL, NULL, strBody.c_str (), vFlvs, FDT_EXACTURLMATCH);
if (!m_vDownloads.empty ())
return S_OK;
FindFlvDownloadInInBetweenDlgs (pTraffic, pWebPage, NULL, NULL, strBody.c_str (), vFlvs, FDT_FILEURLMATCH);
if (!m_vDownloads.empty ())
return S_OK;
FindDialogsByUniqueUrlWordsMatch (strBody.c_str (), vFlvs, vFlvs2);
if (vFlvs2.empty ())
FindDialogsByUniqueUrlWordsMatch2 (pWebPage->strRequestUrl.c_str (), vFlvs, vFlvs2);
if (!vFlvs2.empty ())
{
wstring wstrTitle;
ExtractTitleFromHtml (pWebPage, wstrTitle);
for (size_t i = 0; i < vFlvs2.size (); i++)
AddFlvDownload (pWebPage, vFlvs2 [i], wstrTitle.c_str ());
return S_OK;
}
FindFlvDownloadInInBetweenDlgs (pTraffic, pWebPage, NULL, NULL, strBody.c_str (), vFlvs, FDT_BYUNIQUEWORDS);
if (!m_vDownloads.empty ())
return S_OK;
return m_vDownloads.empty () ? S_FALSE : S_OK;
}
HRESULT vmsHttpFlvTrafficAnalyzer5::FindFlvDownloads(LPCSTR pszHtmlPageUrl, LPCSTR pszFrameUrl, LPCSTR pszSwfUrl, LPCSTR pszFlashVars,
vector <string> &vOtherSwfUrls, vector <string> &vOtherFlashVars, vmsWinSockHttpDlgTree *pDlgTree)
{
LOGFN ("vmsHttpFlvTrafficAnalyzer5::FindFlvDownloads");
LOG ("Web page url: %s", pszHtmlPageUrl);
LOG ("Frame url: %s", pszFrameUrl ? pszFrameUrl : "");
LOG ("Swf url: %s", pszSwfUrl ? pszSwfUrl : "");
assert (vOtherFlashVars.size () == vOtherSwfUrls.size ());
if (vOtherFlashVars.size () != vOtherSwfUrls.size ())
return E_INVALIDARG;
int nWebPageStartIndex = 0, nFrameStartIndex = 0;
HTTPDLGTREEITEM p1stWebPage = NULL;
vector <HTTPDLGTREEITEMLIST> vFlvs;
vFlvs.reserve (50);
#ifdef LOG_WEBFILES_TREE
extern LONG _cInOnGetItBtnClicked;
bool bLog = _cInOnGetItBtnClicked != 0;
if (!bLog)
{
static DWORD _dwLastTicks = 0;
if ((GetKeyState (VK_SHIFT) & GetKeyState (VK_CONTROL) & 0x8000) &&
(GetTickCount () - _dwLastTicks > 10000))
{
bLog = true;
_dwLastTicks = GetTickCount ();
}
}
if (bLog)
{
pDlgTree->LogAllWebFiles ();
pDlgTree->LogAllFlvTrees ();
}
#endif
while (true)
{
HTTPDLGTREEITEM pWebPage = pDlgTree->FindTreeItem (pszHtmlPageUrl,
vmsHttpTrafficCollector::HttpDialog::HTML, false, &nWebPageStartIndex);
if (!pWebPage)
break;
if (!p1stWebPage)
p1stWebPage = pWebPage;
pDlgTree->MakeItWebPageIfNotAlready ((vmsWinSockHttpDlgTree::TreeItem*)pWebPage);
HTTPDLGTREEITEM pFrame = pszFrameUrl && *pszFrameUrl ?
pDlgTree->FindTreeItem (pszFrameUrl, vmsHttpTrafficCollector::HttpDialog::HTML, false, &nFrameStartIndex) : NULL;
LOGHTTPDLGTREE ("Web page tree", pWebPage, 5);
#ifdef LOG_WEBFILES_TREE
if (bLog)
{
pDlgTree->LogTreeItem (pWebPage, 5);
if (pFrame)
pDlgTree->LogTreeItem (pFrame, 5);
static LONG _cCalls = 0;
if (p1stWebPage == pWebPage)
{
LogMainCall (InterlockedIncrement (&_cCalls), pszHtmlPageUrl, pszFrameUrl, pszSwfUrl, pszFlashVars,
vOtherSwfUrls, vOtherFlashVars, pDlgTree, pWebPage, pFrame);
}
}
#endif
vector <HTTPDLGTREEITEMLIST> vFlvs2;
vFlvs2.reserve (50);
pDlgTree->GetChildItemsWithPath (pWebPage, vFlvs2, vmsHttpTrafficCollector::HttpDialog::FLV, 5);
for (size_t iFlv = 0; iFlv < vFlvs2.size (); iFlv++)
vFlvs.push_back (vFlvs2 [iFlv]);
if (pFrame)
{
vFlvs2.clear ();
pDlgTree->GetChildItemsWithPath (pFrame, vFlvs2, vmsHttpTrafficCollector::HttpDialog::FLV, 5);
for (size_t iFlv = 0; iFlv < vFlvs2.size (); iFlv++)
vFlvs.push_back (vFlvs2 [iFlv]);
}
if ((pszFrameUrl == NULL || *pszFrameUrl == 0) &&
(pszSwfUrl == NULL || *pszSwfUrl == 0) &&
(pszFlashVars == NULL || *pszFlashVars == 0))
{
const vmsWinSockHttpDlgTree::WebPageDetails *pwpd = pWebPage->getWebPageDetails ();
if (pwpd)
{
for (size_t i = 0; i < pwpd->vFrameUrls.size (); i++)
{
HTTPDLGTREEITEM pFrame = pDlgTree->FindTreeItem (pwpd->vFrameUrls [i].c_str (),
vmsHttpTrafficCollector::HttpDialog::HTML);
if (pFrame)
{
vFlvs2.clear ();
pDlgTree->GetChildItemsWithPath (pFrame, vFlvs2, vmsHttpTrafficCollector::HttpDialog::FLV, 5);
for (size_t iFlv = 0; iFlv < vFlvs2.size (); iFlv++)
vFlvs.push_back (vFlvs2 [iFlv]);
}
}
}
}
}
if (!p1stWebPage)
return E_INVALIDARG;
pDlgTree->GetChildItemsWithPath_EraseDupes (vFlvs);
vector <HTTPDLGTREEITEMLIST> vFlvsToAdd;
string strFlashVars0;
if (pszSwfUrl && *pszSwfUrl)
{
if (pszFlashVars)
strFlashVars0 = pszFlashVars;
string str;
ExtractFlashVarsFromSwfUrl (pszSwfUrl, NULL, pDlgTree, str);
if (!strFlashVars0.empty ())
strFlashVars0 += '&';
strFlashVars0 += str;
if (!vOtherSwfUrls.empty ())
{
vmsSwfObjectsAndFlvsConformity swfConfs (NULL, pDlgTree);
swfConfs.addSwfObject (pszSwfUrl, pszFlashVars);
for (size_t i = 0; i < vOtherSwfUrls.size (); i++)
swfConfs.addSwfObject (vOtherSwfUrls [i].c_str (), vOtherFlashVars [i].c_str ());
for (size_t i = 0; i < vFlvs.size (); i++)
swfConfs.addFlv (NULL, NULL, &vFlvs [i]);
swfConfs.FindComformity ();
for (size_t i = 0; i < swfConfs.getConformitiesCount (); i++)
{
const vmsSwfObjectsAndFlvsConformity::Conformity *conf = swfConfs.getConformity (i);
if (conf->pSwf->strSwfUrl == pszSwfUrl && conf->pSwf->strFlashVars == pszFlashVars)
vFlvsToAdd.push_back (conf->pFlv->vPathFromWebPageToFlv);
}
}
else
{
vFlvsToAdd = vFlvs;
}
}
else
{
vFlvsToAdd = vFlvs;
}
for (size_t i = 0; i < vFlvsToAdd.size (); i++)
{
wstring wstrTitle;
GetFlvTitle (p1stWebPage->spDlg, strFlashVars0.c_str (), vFlvsToAdd [i], wstrTitle);
assert (vFlvsToAdd [i][vFlvsToAdd [i].size () - 1]->spDlg->enCT == vmsHttpTrafficCollector::HttpDialog::FLV);
AddFlvDownload (p1stWebPage->spDlg, vFlvsToAdd [i][vFlvsToAdd [i].size () - 1]->spDlg, wstrTitle.c_str ());
}
return m_vDownloads.size () ? S_OK : S_FALSE;
}
