Question on validation of mining models

Hi, guys,

Thanks for your kind attention.

Just want to make things perfectly work and make the most of our fantastic SQL Server 2005 Data Mining Engine. Can any of you here give me some super advices on the validation of the mining models. As we always see, the 3 aspects of a mining model are: Score, Population correct, and Predict Probability. So the question is: How can we combine these three aspects to best judge the mining models by being able to tell which model is the best one? And to what extent can we really trust these mining models?

These are very important before we can actually bring the models into work to convince other people who have no ideas what are going on with these models. Yes, we just want to convince them with the results of these models and make the most of them and best help them getting the most from their business operations etc.

By the way please can you explain a bit details on each of these aspects? Thanks again.

I am looking forward to hearing from you shortly and thanks bunch for your help.

With best regards,

Yours sincerely,

Hi, Guys,

I am still waiting for your advices.

Thanks.

With best regards,

Yours sincerely,

|||

I heard about a book "Quality Measures in Data Mining" by Fabrice Guillet and Howard J. Hamilton. It may contain some super advices.

Gigi Ciubuc

www.sqlserver.ro

|||

Hi, Gigi Ciubuc,

Thanks for the imformation. Any other ideas from any other experts here as well?

Thanks.

With best regards,

Yours sincerely,

Question on Report Builder

I am launching the Report Builder from my web application. The launching works fine. The Report Builder displays all the models published on the Report Server, whereas the user launching the report builder doesnt have permission on most of the folders containing the models. The permission set on the folders individually (not inherit from parent) using Management Studio.

The Report Server and the Report Builder folder under that both are running with Integrated Windows Authentication.

Is there anything am missing to check ? I want the user should be able to see only those models, which he/she has been given permission to access.

For info, the web application from where the Report Builder is launched is using Forms Authentication. So, Anonymous Access is enabled. If this is the problem, can I create a separate web page and inside a folder (under website) and apply Integrated Windows Authentication to the folder and launch Report Builder from the web page. Will it work ?

Thanks in advance.

Users will only see the models to which they have been granted the View Models permission (part of the the Browser and Report Builder roles by default), with two exceptions:

1. Members of the local administrators group will always see all models on the Report Server regardless of permission settings.

2. If you launch Report Builder as a given user, then change the group membership of that user and re-launch the application, you will not see any changes in behavior because ClickOnce actually started a background process the first time around called dfsvc.exe, and this process is what actually launches the application both times. Since the background process doesn't get restarted, it is still running with the original security token, and is unaware of the change in group membership. In my experience, if you just kill the dfsvc.exe process and re-launch your application, everything works as expected the next time around.

|||This all sounds great, how do I, from my own website, launch report builder as a known user?|||

Hi,

Did you have luck in launching the report builder from your site with a specific user security? I am also having the same challenge.

Any help will be appreciated.

|||

Rajeeb,

May I ask you a question,

How do you setup the access right to enable the users only see their folders.

say for user A, he can only see Folder A

User B, can only see Folder B.

Folder A and B should be subfolders of Home. I tried to do it. but if I don't give user A the access to Home, it won't show report builder icon on the web page when I log in as user A.

Thanks

Jack

|||Hi Jack,

You can setup access rights as per your example either after you login to Reporting Services using Management Studio or from Report Manager. You have to give permission to both A and B on Home, then in Folder A and B properties, remove the other user (dont follow the option of inheriting access rights from parent folder).

We have already moved on to implement Forms Authentication and handle security our own way. Our web application and report manager works perfectly fine. But, Report Builder using Forms Authentication has so many issues. We havent been successfully implement it yet, its been almost one year now.

I have put many topics on Report Builder using forms authentication but noone replies (just do a search on Report Builder Forms authentication). Its been really frustrating. Imagine after going live with 2 clients on same report server, having problem with Report Builder. We are stuck and nowhere now. We made a big mistake of selling the features of adhoc reports.

My suggestion is, wait for future releases (unless you are in a hurry).

Thanks.
|||

Rajeeb,

Thank you for your reply.

Could you please tell me what right you give user A and B on the Home folder. It seems to me you have to give them content manager right. otherwise the report builder icon won't show on the report manager web page.

Thank you for your help

Jack

Question on Report Builder

I am launching the Report Builder from my web application. The launching works fine. The Report Builder displays all the models published on the Report Server, whereas the user launching the report builder doesnt have permission on most of the folders containing the models. The permission set on the folders individually (not inherit from parent) using Management Studio.

The Report Server and the Report Builder folder under that both are running with Integrated Windows Authentication.

Is there anything am missing to check ? I want the user should be able to see only those models, which he/she has been given permission to access.

For info, the web application from where the Report Builder is launched is using Forms Authentication. So, Anonymous Access is enabled. If this is the problem, can I create a separate web page and inside a folder (under website) and apply Integrated Windows Authentication to the folder and launch Report Builder from the web page. Will it work ?

Thanks in advance.

Users will only see the models to which they have been granted the View Models permission (part of the the Browser and Report Builder roles by default), with two exceptions:

1. Members of the local administrators group will always see all models on the Report Server regardless of permission settings.

2. If you launch Report Builder as a given user, then change the group membership of that user and re-launch the application, you will not see any changes in behavior because ClickOnce actually started a background process the first time around called dfsvc.exe, and this process is what actually launches the application both times. Since the background process doesn't get restarted, it is still running with the original security token, and is unaware of the change in group membership. In my experience, if you just kill the dfsvc.exe process and re-launch your application, everything works as expected the next time around.

|||This all sounds great, how do I, from my own website, launch report builder as a known user?|||

Hi,

Did you have luck in launching the report builder from your site with a specific user security? I am also having the same challenge.

Any help will be appreciated.

|||

Rajeeb,

May I ask you a question,

How do you setup the access right to enable the users only see their folders.

say for user A, he can only see Folder A

User B, can only see Folder B.

Folder A and B should be subfolders of Home. I tried to do it. but if I don't give user A the access to Home, it won't show report builder icon on the web page when I log in as user A.

Thanks

Jack

|||Hi Jack,

You can setup access rights as per your example either after you login to Reporting Services using Management Studio or from Report Manager. You have to give permission to both A and B on Home, then in Folder A and B properties, remove the other user (dont follow the option of inheriting access rights from parent folder).

We have already moved on to implement Forms Authentication and handle security our own way. Our web application and report manager works perfectly fine. But, Report Builder using Forms Authentication has so many issues. We havent been successfully implement it yet, its been almost one year now.

I have put many topics on Report Builder using forms authentication but noone replies (just do a search on Report Builder Forms authentication). Its been really frustrating. Imagine after going live with 2 clients on same report server, having problem with Report Builder. We are stuck and nowhere now. We made a big mistake of selling the features of adhoc reports.

My suggestion is, wait for future releases (unless you are in a hurry).

Thanks.|||

Rajeeb,

Thank you for your reply.

Could you please tell me what right you give user A and B on the Home folder. It seems to me you have to give them content manager right. otherwise the report builder icon won't show on the report manager web page.

Thank you for your help

Jack

Question on Report Builder

I am launching the Report Builder from my web application. The launching works fine. The Report Builder displays all the models published on the Report Server, whereas the user launching the report builder doesnt have permission on most of the folders containing the models. The permission set on the folders individually (not inherit from parent) using Management Studio.

The Report Server and the Report Builder folder under that both are running with Integrated Windows Authentication.

Is there anything am missing to check ? I want the user should be able to see only those models, which he/she has been given permission to access.

For info, the web application from where the Report Builder is launched is using Forms Authentication. So, Anonymous Access is enabled. If this is the problem, can I create a separate web page and inside a folder (under website) and apply Integrated Windows Authentication to the folder and launch Report Builder from the web page. Will it work ?

Thanks in advance.

Users will only see the models to which they have been granted the View Models permission (part of the the Browser and Report Builder roles by default), with two exceptions:

1. Members of the local administrators group will always see all models on the Report Server regardless of permission settings.

2. If you launch Report Builder as a given user, then change the group membership of that user and re-launch the application, you will not see any changes in behavior because ClickOnce actually started a background process the first time around called dfsvc.exe, and this process is what actually launches the application both times. Since the background process doesn't get restarted, it is still running with the original security token, and is unaware of the change in group membership. In my experience, if you just kill the dfsvc.exe process and re-launch your application, everything works as expected the next time around.

|||This all sounds great, how do I, from my own website, launch report builder as a known user?|||

Hi,

Did you have luck in launching the report builder from your site with a specific user security? I am also having the same challenge.

Any help will be appreciated.

|||

Rajeeb,

May I ask you a question,

How do you setup the access right to enable the users only see their folders.

say for user A, he can only see Folder A

User B, can only see Folder B.

Folder A and B should be subfolders of Home. I tried to do it. but if I don't give user A the access to Home, it won't show report builder icon on the web page when I log in as user A.

Thanks

Jack

|||Hi Jack,

You can setup access rights as per your example either after you login to Reporting Services using Management Studio or from Report Manager. You have to give permission to both A and B on Home, then in Folder A and B properties, remove the other user (dont follow the option of inheriting access rights from parent folder).

We have already moved on to implement Forms Authentication and handle security our own way. Our web application and report manager works perfectly fine. But, Report Builder using Forms Authentication has so many issues. We havent been successfully implement it yet, its been almost one year now.

I have put many topics on Report Builder using forms authentication but noone replies (just do a search on Report Builder Forms authentication). Its been really frustrating. Imagine after going live with 2 clients on same report server, having problem with Report Builder. We are stuck and nowhere now. We made a big mistake of selling the features of adhoc reports.

My suggestion is, wait for future releases (unless you are in a hurry).

Thanks.|||

Rajeeb,

Thank you for your reply.

Could you please tell me what right you give user A and B on the Home folder. It seems to me you have to give them content manager right. otherwise the report builder icon won't show on the report manager web page.

Thank you for your help

Jack

Question on dealing with missing values for training models

Hi, all,

Just really wonder what is the good idea to deal with missing values? Should we leave the missing values there in the traning data set ? Or replace it with other values?

What I am really concerned is that if we simply replace those missing values with other values, then how will it really affect the correctness of the training models?

I am looking forward to hearing from you for the above issue and it will be really great if we have any kind of best practices of dealing with this issue.

Thanks.

With best regards,

Yours sincerely,

I have the same question, what's the best practice to replace NULL values in the source data? Should they be left as NULL or replaced with some predetermined default value?|||

For most algorithms, NULL and NULL substituted with a value do not mean the same thing. NULL is usually ignored whereas a substituted NULL is a real state value. The correct modeling approach depends on your data. If NULL has some information in your scenario (e.g. NULL for State means customer from a different country), it's a good idea to sutstitute NULL with a default value before training. If however, NULL means absense of data and has no information, it should be left as is.

Hope this helps.

|||

Hi, Shuvro,

Thanks a lot. It's is quite clear to me now to have a clear idea on dealing with the Null values.

With best regards,

Yours sincerely,

Question on dealing with missing values for training models

Hi, all,

Just really wonder what is the good idea to deal with missing values? Should we leave the missing values there in the traning data set ? Or replace it with other values?

What I am really concerned is that if we simply replace those missing values with other values, then how will it really affect the correctness of the training models?

I am looking forward to hearing from you for the above issue and it will be really great if we have any kind of best practices of dealing with this issue.

Thanks.

With best regards,

Yours sincerely,

I have the same question, what's the best practice to replace NULL values in the source data? Should they be left as NULL or replaced with some predetermined default value?|||

For most algorithms, NULL and NULL substituted with a value do not mean the same thing. NULL is usually ignored whereas a substituted NULL is a real state value. The correct modeling approach depends on your data. If NULL has some information in your scenario (e.g. NULL for State means customer from a different country), it's a good idea to sutstitute NULL with a default value before training. If however, NULL means absense of data and has no information, it should be left as is.

Hope this helps.

|||

Hi, Shuvro,

Thanks a lot. It's is quite clear to me now to have a clear idea on dealing with the Null values.

With best regards,

Yours sincerely,

Question on data mining report performance optimization

Hi, all experts here,

Would any of you give me any ideas for how could we optimize the report on data mining models? (as we know, for the data mining report, we have to select the mining model and the case table)

Hope it is clear for your advices and help.

Thanks a lot in advance and I am looking forward to hearing from you shortly.

With best regards,

Yours sincerely,

What kind of report is this? An accuracy report, such as lift chart or scatter plot?

Also, what kind of algorithm is involved?

|||

Yes, accuracy report and lift chart report. All SQL Server 2005 built-in algorithms there.

Any suggestions?

With best regards,

Yours sincerely,

|||

Service Pack 2 added some performance improvements in generating accuracy reports for the Naive Bayes algorithm.

Besides this, there is not much you can do to improve the performance of these reports.

You could try to use the tip here: http://sqlserverdatamining.com/DMCommunity/TipsNTricks/4566.aspx and implement your own accuracy chart using reporting services

What kind of problems are you running into? Could you please let us know how many test cases are you using and how much time does it take to populate the report?

|||The performance of the lift chart/etc. is directly related to the number of cases you are testing. I.e. if you cut the number of cases in half, you will cut the time in half.|||

Hi, all,

Thanks a lot for your very kind advices and help.

With best regards,

Yours sincerely,

Question on attributes selection for un-supervised algorithms and supervised algorithms

Hi, all,

Thanks for your kind attention.

Just wonder is there any good idea for us to select attributes for training models? Both for non-supervised algorithms like Association Rules and Clustering etc. and supervised algorithms like decision tree etc.

It will be much interesting to hear from you for any best practices and popular methods of dealing with this issue.

I am looking forward to hearing from you and thanks for your advices.

With best regards,

Yours sincerely,

Hi,

I assume that you are trying to select a subset of all your attributes to train the models. SQL Server Data Mining Algorithms have built in feature selection methods. For example, the Microsoft Decision Trees support the following attribute scoring methods: Entropy, Bayesian with K2 Prior and Bayesian Dirichlet Equivalent with Uniform Prior (which is used by default). When feature selection is necessary, the algorithm calculates the scores for each attribute and only train trees with selected features (with top scores, of course). Other algorithms have similar feature selection mechanism.

Thanks,

Question on attributes selection for un-supervised algorithms and supervised algorithms

Hi, all,

Thanks for your kind attention.

Just wonder is there any good idea for us to select attributes for training models? Both for non-supervised algorithms like Association Rules and Clustering etc. and supervised algorithms like decision tree etc.

It will be much interesting to hear from you for any best practices and popular methods of dealing with this issue.

I am looking forward to hearing from you and thanks for your advices.

With best regards,

Yours sincerely,

Hi,

I assume that you are trying to select a subset of all your attributes to train the models. SQL Server Data Mining Algorithms have built in feature selection methods. For example, the Microsoft Decision Trees support the following attribute scoring methods: Entropy, Bayesian with K2 Prior and Bayesian Dirichlet Equivalent with Uniform Prior (which is used by default). When feature selection is necessary, the algorithm calculates the scores for each attribute and only train trees with selected features (with top scores, of course). Other algorithms have similar feature selection mechanism.

Thanks,

Question on association models: MINIMUM_ITEMSET_SIZE

I've been experimenting with the algorithm parameters for a market basket association model. The default MINIMUM_ITEMSET_SIZE is 1. This doesn't seem to make sense: what is the point of a single-member itemset? However changing the value to 2 substantially reduces the proportion of good recommendations obtained (which I'm testing via a holdout approach).

So I'm obviously misunderstanding what the parameter means. Can someone explain it please, and also explain the observation above)?

Every item in a dataset is an "itemset", so there are always "1-itemsets", that's just the list of items and their counts. MINIMUM_ITEMSET_SIZE is really only there for those who are only interested in itemsets larger than 2. Of course, the process of building the model still has to create all the smaller itemsets to get to the larger ones, but it then discards them for prediction and reporting purposes.